Xorg failure (abort/crash) after wake-up from S3 sleep on Nvidia drivers, workaround to Xorg

[mikkorantalainen]

When system is waking up from S3 sleep and Nvidia driver calls nvkms_open() after calling drm_master_open(), the driver fails to handle return value -EINTR. Instead of repeating the call to nvkms_open() (as expected for an interrupted syscall) the driver proceeds to return failure for the Xorg call to EnterVT() instead.

See https://forums.developer.nvidia.com/t/xorg-crashes-on-resume-from-deep-sleep-nvidia-gpu-0-failed-to-acquire-modesetting-permission/380031/9?u=mikko.rantalainen for details.

As I don't have the source code for the driver, I've been debugging this with bpftrace and according to that, the driver never retries call to nvkms_open() after receiving -EINTR and Xorg immediately crashes/aborts as a result after single -EINTR return value from nvkms_open().

[mikkorantalainen]

Additional details, copied from the above mentioned forum post:

Xorg stacktrace:

$ apport-retrace -v -S system -s Xorg-for-retrace.crash
...
--- stack trace ---
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
        tid = <optimized out>
        ret = 0
        pd = <optimized out>
        old_mask = {__val = {0}}
        ret = <optimized out>
        pd = <optimized out>
        old_mask = <optimized out>
        ret = <optimized out>
        tid = <optimized out>
        ret = <optimized out>
        resultvar = <optimized out>
        resultvar = <optimized out>
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
        __futex = <optimized out>
        resultvar = <optimized out>
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
        __futex = <optimized out>
        __private = <optimized out>
        __oldval = <optimized out>
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
No locals.
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
No locals.
#3  0x00007bbdee64527e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
        ret = <optimized out>
#4  0x00007bbdee6288ff in __GI_abort () at ./stdlib/abort.c:79
        save_stage = 1
        act = {__sigaction_handler = {sa_handler = 0x20, sa_sigaction = 0x20}, sa_mask = {__val = {72339069014638592, 62575528714, 62580528714, 100000, 62580528764, 140724979708304, 5000000, 140724979708376, 62580528714, 21474836480000003, 140724979708239, 136029721571328, 136055664410680, 0, 136051746144260, 136055678587696}}, sa_flags = -974429440, sa_restorer = 0x7bbdee8044e0 <_IO_2_1_stderr_>}
#5  0x0000595fdd7e8fae in OsAbort () at ../../../../os/utils.c:1361
No locals.
#6  0x0000595fdd7eeaf7 in AbortServer () at ../../../../os/log.c:879
No locals.
#7  0x0000595fdd7efb77 in FatalError (f=f@entry=0x595fdd7f8767 "EnterVT failed for screen %d\n") at ../../../../os/log.c:1017
        args = {{gp_offset = 16, fp_offset = 48, overflow_arg_area = 0x7ffd166d3410, reg_save_area = 0x7ffd166d3340}}
        args2 = {{gp_offset = 8, fp_offset = 48, overflow_arg_area = 0x7ffd166d3410, reg_save_area = 0x7ffd166d3340}}
        beenhere = 1
#8  0x0000595fdd6a8dd8 in xf86VTEnter () at ../../../../../../hw/xfree86/common/xf86Events.c:457
        i = 0
        pInfo = <optimized out>
        ih = <optimized out>
        i = <optimized out>
        pInfo = <optimized out>
        ih = <optimized out>
#9  xf86VTEnter () at ../../../../../../hw/xfree86/common/xf86Events.c:440
        i = <optimized out>
        pInfo = <optimized out>
        ih = <optimized out>
#10 0x0000595fdd66badd in WakeupHandler (result=result@entry=0) at ../../../../dix/dixutils.c:427
        i = <optimized out>
        j = <optimized out>
#11 0x0000595fdd7df508 in WaitForSomething (are_ready=1) at ../../../../os/WaitFor.c:210
        i = <optimized out>
        timeout = 0
        pollerr = 11
        were_ready = 0
        timer_is_running = 1
#12 0x0000595fdd66686a in Dispatch () at ../../../../dix/dispatch.c:492
        result = <optimized out>
        client = <optimized out>
        start_tick = <optimized out>
#13 0x0000595fdd66aeda in dix_main (argc=11, argv=<optimized out>, envp=<optimized out>) at ../../../../dix/main.c:274
        i = <optimized out>
        alwaysCheckForInput = {0, 1}
#14 0x00007bbdee62a1ca in __libc_start_call_main (main=main@entry=0x595fdd653360 <main>, argc=argc@entry=11, argv=argv@entry=0x7ffd166d36a8) at ../sysdeps/nptl/libc_start_call_main.h:58
        self = <optimized out>
        result = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140724979709608, 796662675305042293, 11, 0, 98268273312088, 136055687385088, 796662675185504629, 256600056618848629}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0xb, 0x7ffd166d36a0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 11}}}
        not_first_call = <optimized out>
#15 0x00007bbdee62a28b in __libc_start_main_impl (main=0x595fdd653360 <main>, argc=11, argv=0x7ffd166d36a8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd166d3698) at ../csu/libc-start.c:360
No locals.
#16 0x0000595fdd653395 in _start ()
No symbol table info available.
--- source code stack trace ---
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
  [Error: pthread_kill.c was not found in source tree]
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
  [Error: pthread_kill.c was not found in source tree]
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
  [Error: pthread_kill.c was not found in source tree]
#3  0x00007bbdee64527e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
  [Error: raise.c was not found in source tree]
#4  0x00007bbdee6288ff in __GI_abort () at ./stdlib/abort.c:79
  [Error: abort.c was not found in source tree]
#5  0x0000595fdd7e8fae in OsAbort () at ../../../../os/utils.c:1361
  1356:     /* abort() raises SIGABRT, so we have to stop handling that to prevent
  1357:      * recursion
  1358:      */
  1359:     OsSignal(SIGABRT, SIG_DFL);
  1360: #endif
  1361:     abort();
  1362: }
  1363: 
  1364: #if !defined(WIN32)
  1365: /*
  1366:  * "safer" versions of system(3), popen(3) and pclose(3) which give up
#6  0x0000595fdd7eeaf7 in AbortServer () at ../../../../os/log.c:879
  874:     OsCleanup(TRUE);
  875:     AbortDevices();
  876:     ddxGiveUp(EXIT_ERR_ABORT);
  877:     fflush(stderr);
  878:     if (CoreDump)
  879:         OsAbort();
  880:     exit(1);
  881: }
  882: 
  883: #define AUDIT_PREFIX "AUDIT: %s: %ld: "
  884: #ifndef AUDIT_TIMEOUT
#7  0x0000595fdd7efb77 in FatalError (f=f@entry=0x595fdd7f8767 "EnterVT failed for screen %d\n") at ../../../../os/log.c:1017
  1012:     if (!beenhere)
  1013:         OsVendorFatalError(f, args2);
  1014:     va_end(args2);
  1015:     if (!beenhere) {
  1016:         beenhere = TRUE;
  1017:         AbortServer();
  1018:     }
  1019:     else
  1020:         OsAbort();
  1021:  /*NOTREACHED*/}
  1022: 
#8  0x0000595fdd6a8dd8 in xf86VTEnter () at ../../../../../../hw/xfree86/common/xf86Events.c:457
  452:     if (xorgHWAccess)
  453:         xf86EnableIO();
  454:     for (i = 0; i < xf86NumScreens; i++) {
  455:         xf86Screens[i]->vtSema = TRUE;
  456:         if (!xf86Screens[i]->EnterVT(xf86Screens[i]))
  457:             FatalError("EnterVT failed for screen %d\n", i);
  458:     }
  459:     for (i = 0; i < xf86NumGPUScreens; i++) {
  460:         xf86GPUScreens[i]->vtSema = TRUE;
  461:         if (!xf86GPUScreens[i]->EnterVT(xf86GPUScreens[i]))
  462:             FatalError("EnterVT failed for gpu screen %d\n", i);
#9  xf86VTEnter () at ../../../../../../hw/xfree86/common/xf86Events.c:440
  435:     }
  436:     input_unlock();
  437: }
  438: 
  439: void
  440: xf86VTEnter(void)
  441: {
  442:     int i;
  443:     InputInfoPtr pInfo;
  444:     IHPtr ih;
  445: 
#10 0x0000595fdd66badd in WakeupHandler (result=result@entry=0) at ../../../../dix/dixutils.c:427
  422:         (*screenInfo.screens[i]->WakeupHandler) (screenInfo.screens[i], result);
  423:     for (i = 0; i < screenInfo.numGPUScreens; i++)
  424:         (*screenInfo.gpuscreens[i]->WakeupHandler) (screenInfo.gpuscreens[i], result);
  425:     for (i = numHandlers - 1; i >= 0; i--)
  426:         if (!handlers[i].deleted)
  427:             (*handlers[i].WakeupHandler) (handlers[i].blockData, result);
  428:     if (handlerDeleted) {
  429:         for (i = 0; i < numHandlers;)
  430:             if (handlers[i].deleted) {
  431:                 for (j = i; j < numHandlers - 1; j++)
  432:                     handlers[j] = handlers[j + 1];
#11 0x0000595fdd7df508 in WaitForSomething (are_ready=1) at ../../../../os/WaitFor.c:210
  205:         if (dispatchException)
  206:             i = -1;
  207:         else
  208:             i = ospoll_wait(server_poll, timeout);
  209:         pollerr = GetErrno();
  210:         WakeupHandler(i);
  211:         if (i <= 0) {           /* An error or timeout occurred */
  212:             if (dispatchException)
  213:                 return FALSE;
  214:             if (i < 0) {
  215:                 if (pollerr != EINTR &amp;&amp; !ETEST(pollerr)) {
#12 0x0000595fdd66686a in Dispatch () at ../../../../dix/dispatch.c:492
  487:         if (InputCheckPending()) {
  488:             ProcessInputEvents();
  489:             FlushIfCriticalOutputPending();
  490:         }
  491: 
  492:         if (!WaitForSomething(clients_are_ready()))
  493:             continue;
  494: 
  495:        /*****************
  496: 	*  Handle events in round robin fashion, doing input between
  497: 	*  each round
#13 0x0000595fdd66aeda in dix_main (argc=11, argv=<optimized out>, envp=<optimized out>) at ../../../../dix/main.c:274
  269: 
  270:         NotifyParentProcess();
  271: 
  272:         InputThreadInit();
  273: 
  274:         Dispatch();
  275: 
  276:         UnrefCursor(rootCursor);
  277: 
  278:         UndisplayDevices();
  279:         DisableAllDevices();
#14 0x00007bbdee62a1ca in __libc_start_call_main (main=main@entry=0x595fdd653360 <main>, argc=argc@entry=11, argv=argv@entry=0x7ffd166d36a8) at ../sysdeps/nptl/libc_start_call_main.h:58
  [Error: libc_start_call_main.h was not found in source tree]
#15 0x00007bbdee62a28b in __libc_start_main_impl (main=0x595fdd653360 <main>, argc=11, argv=0x7ffd166d36a8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd166d3698) at ../csu/libc-start.c:360
  [Error: libc-start.c was not found in source tree]
#16 0x0000595fdd653395 in _start ()

The bpftrace.conf that I used to record traces:

// run with
//    touch output.log
//    tail -F output.log &
//    sudo bpftrace bpftrace.conf >> output.log 2>&1
//
// See https://forums.developer.nvidia.com/t/xorg-crashes-on-resume-from-deep-sleep-nvidia-gpu-0-failed-to-acquire-modesetting-permission/380031/4 for additional info

kprobe:drm_file_alloc
{
    $m = (struct drm_minor *)arg0;
    printf("%llu drm_file_alloc pid=%d comm=%s minor=%d type=%d\n",
           nsecs, pid, comm, $m->index, $m->type);
}

kretprobe:drm_file_alloc
{
    printf("%llu drm_file_alloc EXIT pid=%d comm=%s ret=%p\n",
           nsecs, pid, comm, retval);
}

kprobe:drm_file_free
{
    $f = (struct drm_file *)arg0;
    printf("%llu drm_file_free pid=%d comm=%s id=%llu minor=%d type=%d\n",
           nsecs, pid, comm,
           $f->client_id,
           $f->minor->index,
           $f->minor->type);
}

kprobe:drm_setmaster_ioctl
{
    printf("%llu drm_setmaster_ioctl pid=%d comm=%s\n",
           nsecs, pid, comm);
}

kprobe:drm_dropmaster_ioctl
{
    printf("%llu drm_dropmaster_ioctl pid=%d comm=%s\n",
           nsecs, pid, comm);
}

kprobe:drm_master_open
{
    printf("%llu drm_master_open ENTER pid=%d comm=%s\n",
           nsecs, pid, comm);
}

kretprobe:drm_master_open
{
    printf("%llu drm_master_open EXIT pid=%d comm=%s ret=%d\n",
           nsecs, pid, comm, retval);
}

kprobe:nv_drm_master_set
{
    printf("%llu nv_drm_master_set ENTER pid=%d comm=%s\n",
           nsecs, pid, comm);
}

kprobe:nv_drm_grant_permission_ioctl
{
    $p = (uint32 *)arg1;
    @ptype[tid] = $p[2];

    printf("%llu nv_drm_grant_permission_ioctl ENTER pid=%d comm=%s fd=%d dpyId=0x%x type=%u\n",
           nsecs, pid, comm, $p[0], $p[1], $p[2]);
}

kretprobe:nv_drm_grant_permission_ioctl
{
    printf("%llu nv_drm_grant_permission_ioctl EXIT pid=%d comm=%s type=%u ret=%d\n",
           nsecs, pid, comm, @ptype[tid], retval);
    delete(@ptype[tid]);
}

kprobe:pm_vt_switch_required
{
    printf("%llu pm_vt_switch_required ENTER pid=%d comm=%s\n",
           nsecs, pid, comm);
}

kretprobe:pm_vt_switch_required
{
    printf("%llu pm_vt_switch_required EXIT  pid=%d comm=%s ret=%d\n",
           nsecs, pid, comm, retval);
}


kprobe:pm_prepare_console
{
    printf("%llu pm_prepare_console ENTER pid=%d comm=%s\n",
           nsecs, pid, comm);
}

kretprobe:pm_prepare_console
{
    printf("%llu pm_prepare_console EXIT  pid=%d comm=%s ret=%d\n",
           nsecs, pid, comm, retval);
}

kprobe:pm_restore_console
{
    @resume_start = nsecs;
    printf("%llu pm_restore_console ENTER pid=%d comm=%s\n",
           nsecs, pid, comm);
}

kretprobe:pm_restore_console
{
    printf("%llu pm_restore_console EXIT  pid=%d comm=%s ret=%d\n",
           nsecs, pid, comm, retval);
}

kprobe:vt_move_to_console
{
    printf("%llu vt_move_to_console ENTER pid=%d comm=%s target=%d alloc=%d\n",
           nsecs, pid, comm, arg0, arg1);
}

kretprobe:vt_move_to_console
{
    printf("%llu vt_move_to_console EXIT pid=%d comm=%s ret=%d\n",
           nsecs, pid, comm, retval);
}

kprobe:drm_ioctl_kernel
/(arg3 & 2)/
{
    @func[tid] = arg1;
    @flags[tid] = arg3;

    printf("%llu drm_ioctl_kernel ENTER pid=%d comm=%s func=%s flags=0x%x\n",
           nsecs, pid, comm, ksym(arg1), arg3);
}

kretprobe:drm_ioctl_kernel
/@func[tid]/
{
    printf("%llu drm_ioctl_kernel EXIT  pid=%d comm=%s func=%s flags=0x%x ret=%d\n",
           nsecs, pid, comm, ksym(@func[tid]), @flags[tid], retval);

    delete(@func[tid]);
    delete(@flags[tid]);
}

kprobe:drm_master_release
{
    printf("%llu drm_master_release ENTER pid=%d comm=%s\n",
           nsecs, pid, comm);
}

kprobe:nvkms_open
{
    printf("%llu nvkms_open ENTER pid=%d comm=%s\n",
           nsecs, pid, comm);
}

kretprobe:nvkms_open
{
    printf("%llu nvkms_open EXIT pid=%d comm=%s ret=%d\n",
           nsecs, pid, comm, retval);
}

kprobe:nvkms_unlocked_ioctl
/@resume_start && nsecs - @resume_start < 500000000/
{
    $inner = *uptr((uint32 *)arg2);
    @nvkms_cmd[tid] = $inner + 1;

    printf("%llu nvkms_ioctl ENTER pid=%d tid=%d comm=%s inner=%u\n",
           nsecs, pid, tid, comm, $inner);
}

kretprobe:nvkms_unlocked_ioctl
/@nvkms_cmd[tid]/
{
    $inner = @nvkms_cmd[tid] - 1;

    printf("%llu nvkms_unlocked_ioctl EXIT pid=%d tid=%d comm=%s inner=%u ret=%d\n",
           nsecs, pid, tid, comm, $inner, retval);

    delete(@nvkms_cmd[tid]);
}

kprobe:nvkms_close
{
    printf("%llu nvkms_close ENTER pid=%d comm=%s\n",
           nsecs, pid, comm);
}

Successful S3 wake-up (Xorg keeps running as expected):

419178119202504 pm_restore_console ENTER pid=950824 comm=systemd-sleep
419178119206453 vt_move_to_console ENTER pid=950824 comm=systemd-sleep target=6 alloc=0
419178120723064 vt_move_to_console EXIT pid=950824 comm=systemd-sleep ret=62
419178120726048 pm_restore_console EXIT  pid=950824 comm=systemd-sleep ret=0
419178122146622 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=38
419178122150413 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=38 ret=-1
419178122174876 drm_file_alloc pid=945304 comm=Xorg minor=128 type=2
419178122187452 drm_file_alloc EXIT pid=945304 comm=Xorg ret=0xffff8c238f20e000
419178122195292 drm_file_free pid=945304 comm=Xorg id=11528 minor=128 type=2
419178122230145 drm_file_alloc pid=945304 comm=Xorg minor=1 type=0
419178122233136 drm_file_alloc EXIT pid=945304 comm=Xorg ret=0xffff8c238f20e000
419178122233810 drm_master_open ENTER pid=945304 comm=Xorg
419178122239327 nv_drm_master_set ENTER pid=945304 comm=Xorg
419178122239914 drm_master_open EXIT pid=945304 comm=Xorg ret=0
419178122273510 nvkms_open ENTER pid=945304 comm=Xorg
419178122275532 nvkms_open EXIT pid=945304 comm=Xorg ret=0
419178122276759 drm_ioctl_kernel ENTER pid=945304 comm=Xorg func=nv_drm_grant_permission_ioctl flags=0x2
419178122278132 nv_drm_grant_permission_ioctl ENTER pid=945304 comm=Xorg fd=24 dpyId=0x0 type=3
419178146292285 nv_drm_grant_permission_ioctl EXIT pid=945304 comm=Xorg type=3 ret=0
419178146297610 drm_ioctl_kernel EXIT  pid=945304 comm=Xorg func=nv_drm_grant_permission_ioctl flags=0x2 ret=0
419178146307939 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=41
419178146313904 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=41 ret=0
419178146315341 nvkms_close ENTER pid=945304 comm=Xorg
419178146319827 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=4
419178146322010 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=4 ret=0
419178146322539 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=4
419178146322864 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=4 ret=0
419178146323327 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=4
419178146323617 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=4 ret=0
419178146328852 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=6
419178146522330 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=6 ret=0
419178146552880 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=6
419178146560622 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=6 ret=0
419178146569356 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=6
419178146881967 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=6 ret=0
419178146890636 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=6
419178146895781 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=6 ret=0
419178146904228 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=6
419178147249400 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=6 ret=0
419178147258222 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=6
419178148076673 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=6 ret=0
419178148089301 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=6
419178151302099 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=6 ret=0
419178151315788 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=46
419178151316698 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=46 ret=0
419178157672573 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178161877069 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178161895370 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178164817878 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178164826897 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178167750083 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178167761600 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178170338742 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178170347492 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178174236825 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178174262767 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178178062372 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178178091966 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178181557563 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178181574423 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178184768743 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178184781068 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178188006267 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178188020392 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178191040217 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178191055799 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178195096803 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178195108097 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178198545156 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178198557563 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178202520416 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178202546485 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178206237604 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178206257492 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178209645110 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178209734812 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178213379943 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178213498532 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178216595186 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178216704316 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178219805825 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178219897762 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178223515538 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178223605628 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178227461044 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178227522036 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178230913503 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178230963337 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178234699170 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178234706363 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178237749435 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178238927415 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=9
419178273664562 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=9 ret=0
419178273680886 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=59
419178273872082 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=59 ret=0
419178273879567 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=12
419178273893884 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=12 ret=0
419178275988054 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=18
419178279494658 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=18 ret=0
419178279533199 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=17
419178279622619 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=17 ret=0
419178279713435 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178279881230 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178280539845 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178280542681 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178389055889 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178389380341 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178389445515 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178389453407 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178389520294 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178389710390 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178389734800 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178389739959 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178389799565 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178389805914 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178392576498 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178392860739 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178392901292 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178392907892 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178392960961 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178393147207 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178393178914 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178393184289 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178393247774 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178393254018 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178393274763 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178393280742 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178393419139 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178393605773 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178393633202 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178393638469 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178398728917 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398737934 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398742371 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398744191 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398745823 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398747087 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398748574 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398749767 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398751183 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398752263 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398753737 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398754821 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398756247 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398757195 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398758580 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398759518 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398760880 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398762122 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398763494 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398764431 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398765823 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398766778 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398768150 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398769188 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398770596 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398771774 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398773129 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398774361 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398775753 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398776705 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398778240 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398779188 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398780559 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398781508 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398782990 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398784180 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398785533 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398786684 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398788068 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398789024 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398790433 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398791487 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398792862 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398793815 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398795195 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398796253 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398797667 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398798637 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398800048 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398801314 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398802726 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398803832 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398805355 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398806581 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398807971 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398809066 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398810573 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398811515 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398812896 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398813956 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398815378 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398816318 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398817695 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398818784 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398820147 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398821240 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398822633 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398823821 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398825197 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398826133 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398827501 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398829623 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178398831024 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=32
419178398832136 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=32 ret=0
419178404098348 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178404374113 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178404402042 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178404409539 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178405312875 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178405517343 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178406486966 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178406730263 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178406751811 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178406754517 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178408858185 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178409105277 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178410717933 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178410933593 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178413737742 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178413972573 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178421955923 nvkms_ioctl ENTER pid=948639 tid=948639 comm=chrome inner=21
419178421977639 nvkms_unlocked_ioctl EXIT pid=948639 tid=948639 comm=chrome inner=21 ret=0
419178422236615 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178422526367 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178422562748 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178422576061 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178425576731 nvkms_open ENTER pid=945304 comm=Xorg
419178425590994 nvkms_open EXIT pid=945304 comm=Xorg ret=0
419178425594631 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=19
419178425599536 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=19 ret=0
419178425651643 nvkms_ioctl ENTER pid=948639 tid=948639 comm=chrome inner=20
419178425659996 nvkms_unlocked_ioctl EXIT pid=948639 tid=948639 comm=chrome inner=20 ret=0
419178425663117 nvkms_close ENTER pid=948639 comm=chrome
419178425958880 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178426227072 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178438214926 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178438481505 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178438556934 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178438742748 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178439346141 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178439612824 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178441464771 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178441689536 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178442072582 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178442284541 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178447734196 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178447743867 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178448260505 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178448550085 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178448917446 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178448931056 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178449137432 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=10
419178449142762 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=10 ret=0
419178449144520 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=11
419178449146955 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=11 ret=0
419178449147599 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=10
419178449150688 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=10 ret=0
419178449170065 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178449357393 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178454476672 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178454676712 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178456876512 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178457084490 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178459213737 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178459400822 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178462725732 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178462986009 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178463002845 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178463231429 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178481045006 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178481323837 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178481936129 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178482474161 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178483410633 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178483642122 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178484459970 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178484707377 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178581521938 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178581823474 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178590343372 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178590626483 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178592319800 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178592605650 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178592648402 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178592651135 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178592697076 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178592945935 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178592961812 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178592962824 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178593137867 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178593349910 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178593380856 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178593382314 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178593533358 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178593814180 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178594255989 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178594520345 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178594542011 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178594543782 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178594594220 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178594803039 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178594895798 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178595099365 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178595115288 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178595116290 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178595199328 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178595408232 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178595578470 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178595791107 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178595810766 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178595811844 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178596028436 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178596285052 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178596382933 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178596384344 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178596406138 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178596407125 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178596417173 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178596687180 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178596743824 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178596744844 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178596753697 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178597018948 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178597035985 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178597037656 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178597056218 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178597238833 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178597254720 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178597256340 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178597272180 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178597273009 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178597287494 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178597288635 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178597355221 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178597638824 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178597662102 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178597663091 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178597722525 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178597905860 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178597922377 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=50
419178597923554 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=50 ret=0
419178600853640 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178601207504 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178601649797 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178601834434 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178603710797 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178604016751 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0
419178609631159 nvkms_ioctl ENTER pid=945304 tid=945304 comm=Xorg inner=14
419178609896492 nvkms_unlocked_ioctl EXIT pid=945304 tid=945304 comm=Xorg inner=14 ret=0

Failure during wake-up from S3 sleep (Xorg aborts with the above stack trace):

418611018401296 pm_restore_console ENTER pid=944534 comm=systemd-sleep
418611018408246 vt_move_to_console ENTER pid=944534 comm=systemd-sleep target=6 alloc=0
418611018434250 vt_move_to_console EXIT pid=944534 comm=systemd-sleep ret=62
418611018434825 pm_restore_console EXIT  pid=944534 comm=systemd-sleep ret=0
418611025375365 nvkms_ioctl ENTER pid=789619 tid=789619 comm=Xorg inner=38
418611029958066 nvkms_unlocked_ioctl EXIT pid=789619 tid=789619 comm=Xorg inner=38 ret=-4
418611029969617 nvkms_ioctl ENTER pid=789619 tid=789619 comm=Xorg inner=38
418611034932259 nvkms_ioctl ENTER pid=793657 tid=793657 comm=chrome inner=21
418611034964608 nvkms_unlocked_ioctl EXIT pid=789619 tid=789619 comm=Xorg inner=38 ret=-4
418611034987227 nvkms_ioctl ENTER pid=789619 tid=789619 comm=Xorg inner=38
418611039965686 nvkms_unlocked_ioctl EXIT pid=789619 tid=789619 comm=Xorg inner=38 ret=-4
418611039988132 nvkms_ioctl ENTER pid=789619 tid=789619 comm=Xorg inner=38
418611044957439 nvkms_unlocked_ioctl EXIT pid=789619 tid=789619 comm=Xorg inner=38 ret=-4
418611044964396 nvkms_ioctl ENTER pid=789619 tid=789619 comm=Xorg inner=38
418611049967134 nvkms_unlocked_ioctl EXIT pid=789619 tid=789619 comm=Xorg inner=38 ret=-4
418611049983441 nvkms_ioctl ENTER pid=789619 tid=789619 comm=Xorg inner=38
418611054955838 nvkms_unlocked_ioctl EXIT pid=789619 tid=789619 comm=Xorg inner=38 ret=-4
418611054960652 nvkms_ioctl ENTER pid=789619 tid=789619 comm=Xorg inner=38
418611059967011 nvkms_unlocked_ioctl EXIT pid=789619 tid=789619 comm=Xorg inner=38 ret=-4
418611059990136 nvkms_ioctl ENTER pid=789619 tid=789619 comm=Xorg inner=38
418611063705021 nvkms_unlocked_ioctl EXIT pid=793657 tid=793657 comm=chrome inner=21 ret=0
418611063716152 nvkms_unlocked_ioctl EXIT pid=789619 tid=789619 comm=Xorg inner=38 ret=-1
418611063771660 drm_file_alloc pid=789619 comm=Xorg minor=128 type=2
418611063781228 drm_file_alloc EXIT pid=789619 comm=Xorg ret=0xffff8c23834eb600
418611063789720 drm_file_free pid=789619 comm=Xorg id=11500 minor=128 type=2
418611063819442 drm_file_alloc pid=789619 comm=Xorg minor=1 type=0
418611063822026 drm_file_alloc EXIT pid=789619 comm=Xorg ret=0xffff8c23834eb600
418611063822606 drm_master_open ENTER pid=789619 comm=Xorg
418611063827343 nv_drm_master_set ENTER pid=789619 comm=Xorg
418611063827765 drm_master_open EXIT pid=789619 comm=Xorg ret=0
418611063859925 nvkms_open ENTER pid=789619 comm=Xorg
418611064970032 nvkms_open EXIT pid=789619 comm=Xorg ret=-4
418611064988738 drm_file_free pid=789619 comm=Xorg id=11501 minor=1 type=0
418611064993558 drm_master_release ENTER pid=789619 comm=Xorg
418611068994033 nvkms_ioctl ENTER pid=789619 tid=789619 comm=Xorg inner=39
418611068998683 nvkms_unlocked_ioctl EXIT pid=789619 tid=789619 comm=Xorg inner=39 ret=-1
418611276646333 nvkms_ioctl ENTER pid=789619 tid=789619 comm=Xorg inner=9
418611276826442 nvkms_unlocked_ioctl EXIT pid=789619 tid=789619 comm=Xorg inner=9 ret=-1
418611305989847 nvkms_ioctl ENTER pid=789619 tid=789619 comm=Xorg inner=39
418611306002941 nvkms_unlocked_ioctl EXIT pid=789619 tid=789619 comm=Xorg inner=39 ret=-1
418620181392591 drm_file_free pid=789619 comm=InputThread id=10727 minor=128 type=2
418620181671724 nvkms_close ENTER pid=789619 comm=InputThread
418620181691823 drm_file_free pid=789619 comm=InputThread id=10724 minor=128 type=2
418620181756384 nvkms_close ENTER pid=789619 comm=InputThread
418620222003900 drm_file_free pid=793657 comm=HangWatcher id=10738 minor=128 type=2
418620224119410 nvkms_close ENTER pid=793777 comm=chrome
418620224134567 nvkms_close ENTER pid=793777 comm=chrome
418620224135817 drm_file_free pid=793777 comm=chrome id=10735 minor=128 type=2
418620231304972 nvkms_close ENTER pid=793777 comm=chrome

Note that the Nvidia driver correctly repeats calls to nvkms_ioctl inner=38 after receiving -EINTR but it fails to do the same for nvkms_open() and fails after receiving -EINTR causing Xorg to abort as a consequence.

I'm using package linux-lowlatency-hwe-24.04-edge for my kernel image which may increase the probability for receiving -EINTR for a syscall.

[mikkorantalainen]

Reminder: this is a race condition and doesn't happen during every S3 sleep cycle. It appears to happen only after nvkms_open() returns -EINTR.

EnterVT() must try much harder to complete the required syscall because failure to do this results in crashing the whole user desktop session and causes data loss to all currently running programs without unsaved data.

[mikkorantalainen]

According to ChatGPT 5.6, the driver code may actually run open("/dev/nvidia-modeset") which internally calls nvkms_open() and the retry logic could be implemented for errno == EINTR from that open() call only. I couldn't verify if this is a real finding or AI hallucination.

[mikkorantalainen]

Here's the workaround Xorg that I'm currently using on my local system until Nvidia hopefully fixes their driver. This patch may (I haven't yet seen above mentioned failure) be able to workaround the missing EINTR handling in Nvidia driver:

diff -ru xorg-server-21.1.12.original/debian/changelog xorg-server-21.1.12-1ubuntu1.6+nvidia/debian/changelog
--- xorg-server-21.1.12.original/debian/changelog	2026-08-30 22:13:27.000000000 +0300
+++ xorg-server-21.1.12-1ubuntu1.6+nvidia/debian/changelog	2026-08-30 22:36:11.468084209 +0300
@@ -1,3 +1,11 @@
+xorg-server (2:21.1.12-1ubuntu1.6+nvidia-workaround1) UNRELEASED; urgency=medium
+
+  * Automatically repeat call to EnterVT() up to 10 times after Nvidia driver
+    potentially fails for unhandled EINTR. See following for details:
+    https://github.com/canonical/nvidia-graphics-drivers/issues/92
+
+ -- Mikko Rantalainen <mikko.rantalainen@iki.fi>  Sun, 30 Aug 2026 22:34:46 +0300
+
 xorg-server (2:21.1.12-1ubuntu1.6) noble; urgency=medium
 
   * Fix dirty updates for sw rotation (LP: #2146867)
diff -ru xorg-server-21.1.12.original/hw/xfree86/common/xf86Events.c xorg-server-21.1.12-1ubuntu1.6+nvidia/hw/xfree86/common/xf86Events.c
--- xorg-server-21.1.12.original/hw/xfree86/common/xf86Events.c	2024-04-03 23:50:12.000000000 +0300
+++ xorg-server-21.1.12-1ubuntu1.6+nvidia/hw/xfree86/common/xf86Events.c	2026-08-30 22:20:26.966004321 +0300
@@ -456,10 +456,33 @@
         if (!xf86Screens[i]->EnterVT(xf86Screens[i]))
             FatalError("EnterVT failed for screen %d\n", i);
     }
-    for (i = 0; i < xf86NumGPUScreens; i++) {
-        xf86GPUScreens[i]->vtSema = TRUE;
-        if (!xf86GPUScreens[i]->EnterVT(xf86GPUScreens[i]))
-            FatalError("EnterVT failed for gpu screen %d\n", i);
+
+#define ENTER_VT_RETRIES 10
+#define ENTER_VT_RETRY_USEC 100000
+
+    for (i = 0; i < xf86NumScreens; i++) {
+        int attempt;
+
+        xf86Screens[i]->vtSema = TRUE;
+
+        for (attempt = 0; attempt < ENTER_VT_RETRIES; attempt++) {
+            if (xf86Screens[i]->EnterVT(xf86Screens[i]))
+                break;
+
+            if (attempt + 1 == ENTER_VT_RETRIES)
+                FatalError("EnterVT failed for screen %d after %d attempts\n",
+                           i, ENTER_VT_RETRIES);
+
+            xf86Msg(X_WARNING,
+                    "EnterVT failed for screen %d, retrying in %d ms "
+                    "(attempt %d/%d)\n",
+                    i,
+                    ENTER_VT_RETRY_USEC / 1000,
+                    attempt + 1,
+                    ENTER_VT_RETRIES);
+
+            usleep(ENTER_VT_RETRY_USEC);
+        }
     }
     for (i = 0; i < xf86NumScreens; i++) {
         if (xf86Screens[i]->EnableDisableFBAccess)

[mikkorantalainen]

The above patch caused OpenGL subsystem to hang after S3 sleep because it erraneously modified the loop for xf86GPUScreens instead of xf86Screens like it should have done. Here's the fixed patch:

diff -ru xorg-server-21.1.12.original/hw/xfree86/common/xf86Events.c xorg-server-21.1.12/hw/xfree86/common/xf86Events.c
--- xorg-server-21.1.12.original/hw/xfree86/common/xf86Events.c	2024-04-03 23:50:12.000000000 +0300
+++ xorg-server-21.1.12/hw/xfree86/common/xf86Events.c	2026-08-31 11:19:43.665715255 +0300
@@ -451,11 +451,35 @@
 
     if (xorgHWAccess)
         xf86EnableIO();
+
+    #define ENTER_VT_RETRIES 10
+    #define ENTER_VT_RETRY_USEC 100000
+
     for (i = 0; i < xf86NumScreens; i++) {
+        int attempt;
+
         xf86Screens[i]->vtSema = TRUE;
-        if (!xf86Screens[i]->EnterVT(xf86Screens[i]))
-            FatalError("EnterVT failed for screen %d\n", i);
+
+        for (attempt = 0; attempt < ENTER_VT_RETRIES; attempt++) {
+        if (xf86Screens[i]->EnterVT(xf86Screens[i]))
+            break;
+
+        if (attempt + 1 == ENTER_VT_RETRIES)
+            FatalError("EnterVT failed for screen %d after %d attempts\n",
+                       i, ENTER_VT_RETRIES);
+
+        xf86Msg(X_WARNING,
+                "EnterVT failed for screen %d, retrying in %d ms "
+                "(attempt %d/%d)\n",
+                i,
+                ENTER_VT_RETRY_USEC / 1000,
+                attempt + 1,
+                ENTER_VT_RETRIES);
+
+        usleep(ENTER_VT_RETRY_USEC);
+        }
     }
+
     for (i = 0; i < xf86NumGPUScreens; i++) {
         xf86GPUScreens[i]->vtSema = TRUE;
         if (!xf86GPUScreens[i]->EnterVT(xf86GPUScreens[i]))

If you want to try this patch on Ubuntu/Debian system, following should be good enough:

mkdir test-nvidia-workaround
cd test-nvidia-workaround
apt source xserver-xorg-core
cd xorg-server-*
patch -p1 < .../path-to-above-patch.diff
dch --local +retry-EnterVT
debuild -uc -us
cd ..
sudo dpkg -i ./xserver-xorg-core_*.deb

Note that you may need to run apt download xserver-xorg-core && sudo dpkg -i *.deb in an empty directory to restore the official version because the modified version might be considered a newer one by default which requires downgrading to official package. And in case something goes seriously bad, you should figure out before installing the modified version how to restore the previous state if your graphical desktop fails to run at all.

The debuild step took a bit over 2 minutes on my system (i5-13600K).

[mikkorantalainen]

The above corrected patch couldn't prevent Xorg crash today. I gave the crash debug output to ChatGPT 5.6 Sol with high thinking effort and it said following:

The retry loop itself works, but re-calling NVIDIA’s EnterVT() is not equivalent to starting a fresh VT-entry attempt. After the first nvkms_open() -> -EINTR, subsequent calls skip the ownership-acquisition path entirely and go straight into mode-setting, where they fail for lack of permission.

The first EnterVT() attempt follows exactly the failure mechanism we already identified: Xorg repeatedly gets -EINTR from GRAB_OWNERSHIP, eventually gets the expected -EPERM, acquires DRM master successfully, then the temporary nvkms_open() returns -EINTR. It releases DRM master afterward.

But the crucial new observation is what happens on attempts 2–10: They do not repeat the permission-acquisition sequence.

There is no second:

inner=38                    GRAB_OWNERSHIP
drm_file_alloc(card1)
drm_master_open
nvkms_open
nv_drm_grant_permission_ioctl
inner=41                    ACQUIRE_PERMISSIONS

Instead, after the first failed attempt, Xorg/NVIDIA immediately starts issuing mode-setting operations (inner=9) and related operations, eventually getting permission failures. The same pattern repeats throughout the remainder of the retries.

So the NVIDIA DDX behaves roughly like this:

First EnterVT():

    try GRAB_OWNERSHIP
    ↓
    acquire DRM master
    ↓
    open temporary /dev/nvidia-modeset
    ↓
    EINTR
    ↓
    cleanup DRM master
    ↓
    return FALSE

Second EnterVT():

    [does NOT reacquire permissions]
    ↓
    tries to set mode
    ↓
    "Insufficient permissions"
    ↓
    return FALSE

Third EnterVT():

    same thing
    ...

That means the failed first EnterVT() changes some closed NVIDIA userspace/DDX state such that another invocation of EnterVT() is not treated as a fresh entry operation.

Also, based on timestamps the sleeping between retry attempts doesn't work for some reason. Maybe the usleep() returns EINTR, too, but the patch doesn't check for that either? AI computed average waiting time between attempts to be 35 ms and all 10 attemts failed in total of 297 ms. I think that was still long enough for retry logic to work if it were going to workaround the issue in any case.

[mikkorantalainen]

I'm now running following patch which blindly attempts to call LeaveVT() after EnterVT() failure in an attempt to make Nvidia driver to do full EnterVT attempt again from scratch. It appears that Xorg on my system crashes maybe 2 times per 4 days with my usage so I expect to see what happens with this patch in a day or two.

The line xf86Screens[i]->vtSema = FALSE; is used to avoid Xorg calling LeaveVT() again during abort sequence because we have already called it.

diff --git a/hw/xfree86/common/xf86Events.c b/hw/xfree86/common/xf86Events.c
--- a/hw/xfree86/common/xf86Events.c
+++ b/hw/xfree86/common/xf86Events.c
@@ -60,5 +60,6 @@
 #include <X11/Xproto.h>
 #include <X11/Xatom.h>
+#include <errno.h>
 #include "misc.h"
 #include "xf86.h"
 #include "xf86Priv.h"
@@ -450,10 +451,54 @@ xf86VTEnter(void)
     if (xorgHWAccess)
         xf86EnableIO();
+
+#define ENTER_VT_RETRIES 10
+#define ENTER_VT_RETRY_USEC 100000
+
     for (i = 0; i < xf86NumScreens; i++) {
-        xf86Screens[i]->vtSema = TRUE;
-        if (!xf86Screens[i]->EnterVT(xf86Screens[i]))
-            FatalError("EnterVT failed for screen %d\n", i);
+        int attempt;
+        Bool entered = FALSE;
+
+        for (attempt = 0; attempt < ENTER_VT_RETRIES; attempt++) {
+            xf86Screens[i]->vtSema = TRUE;
+
+            if (xf86Screens[i]->EnterVT(xf86Screens[i])) {
+                entered = TRUE;
+                break;
+            }
+
+            /*
+             * FatalError() would cause ddxGiveUp() to call LeaveVT()
+             * for a screen whose vtSema is TRUE anyway.  Do that
+             * cleanup explicitly before retrying EnterVT().
+             */
+            xf86Screens[i]->LeaveVT(xf86Screens[i]);
+            xf86Screens[i]->vtSema = FALSE;
+
+            if (attempt + 1 == ENTER_VT_RETRIES)
+                break;
+
+            xf86Msg(X_WARNING,
+                    "EnterVT failed for screen %d; LeaveVT completed, "
+                    "retrying in %d ms (attempt %d/%d)\n",
+                    i,
+                    ENTER_VT_RETRY_USEC / 1000,
+                    attempt + 1,
+                    ENTER_VT_RETRIES);
+
+            /*
+             * usleep() itself may be interrupted by the same signal
+             * activity involved in the NVIDIA failure.  Do not let
+             * EINTR shorten the retry delay.
+             */
+            while (usleep(ENTER_VT_RETRY_USEC) < 0 && errno == EINTR)
+                ;
+        }
+
+        if (!entered)
+            FatalError("EnterVT failed for screen %d after %d attempts\n",
+                       i, ENTER_VT_RETRIES);
     }
+
     for (i = 0; i < xf86NumGPUScreens; i++) {
         xf86GPUScreens[i]->vtSema = TRUE;
         if (!xf86GPUScreens[i]->EnterVT(xf86GPUScreens[i]))

And this patch will potentially sleep longer than intended if `usleep()` returns after e.g. 60 ms sleep due EINTR. In that case the retry logic will sleep for another 100 ms making the total sleep time for one retry 160 ms in that case. I would expect the `usleep()` not to fail multiple times in sequence so the sleep time between attempts should be max 200 ms and that would result in total failover in 2 seconds if all 10 attempts resulted in 200 ms sleep. If things still fail after that, the Xorg will still abort similar to the case without this patch.

[mikkorantalainen]

Another update: I wouldn't recommend above patch. It seemed to enter some kind of livelock state when EnterVT() failed and the Xorg log ended with

[ 72343.706] (EE) NVIDIA(GPU-0): Failed to acquire modesetting permission.
[ 72343.707] (WW) NVIDIA(0): Failed to set the display configuration
[ 72343.707] (WW) NVIDIA(0):  - Setting a mode on head 0 failed: Insufficient permissions
[ 72343.707] (WW) NVIDIA(0):  - Setting a mode on head 1 failed: Insufficient permissions
[ 72343.707] (WW) NVIDIA(0):  - Setting a mode on head 2 failed: Insufficient permissions
[ 72343.707] (WW) NVIDIA(0):  - Setting a mode on head 3 failed: Insufficient permissions
[ 72343.707] (WW) EnterVT failed for screen 0; LeaveVT completed, retrying in 100 ms (attempt 1/10)

The screen remained fully black after S3 wake-up and I ended up blindly switching to VT1 (Alt+SysRT+R followed by Ctrl+Alt+F1) and rebooting the system (Ctrl+Alt+Delete in VT1) to recover the system. The system appeared to reboot normally so I would assume the Xorg exited normally after receiving either SIGHUP or SIGTERM from the system during reboot sequence.

I'm trying another patch soon to figure out if the issue is while-usleep() loop (if it repeatedly fails with EINTR, it might cause live-lock) or if EnterVT() hanged after LeaveVT() was called after previous failed EnterVT() call.

[mikkorantalainen]

Okay, I think I found a working workaround. It appears that with Nvidia drivers, once EnterVT() call fails, the situation can be fixed by blindly calling LeaveVT() and then waiting a bit and calling EnterVT() again. Repeat until it works.

Here's the patch (I hereby declare it dual licensed as MIT and Xorg default license so it should be okay to include if you want it):

$ cat nvidia-workaround-failed-entervt-v2.patch
diff --git a/hw/xfree86/common/xf86Events.c b/hw/xfree86/common/xf86Events.c
--- a/hw/xfree86/common/xf86Events.c
+++ b/hw/xfree86/common/xf86Events.c
@@ -60,5 +60,6 @@
 #include <X11/Xproto.h>
 #include <X11/Xatom.h>
+#include <time.h>
 #include "misc.h"
 #include "xf86.h"
 #include "xf86Priv.h"
@@ -450,10 +451,78 @@ xf86VTEnter(void)
     if (xorgHWAccess)
         xf86EnableIO();
+
+#define ENTER_VT_RETRIES 10
+#define ENTER_VT_RETRY_USEC 100000
+
     for (i = 0; i < xf86NumScreens; i++) {
-        xf86Screens[i]->vtSema = TRUE;
-        if (!xf86Screens[i]->EnterVT(xf86Screens[i]))
-            FatalError("EnterVT failed for screen %d\n", i);
+        int attempt;
+        Bool entered = FALSE;
+
+        for (attempt = 0; attempt < ENTER_VT_RETRIES; attempt++) {
+            xf86Screens[i]->vtSema = TRUE;
+
+            xf86Msg(X_WARNING,
+                    "Starting EnterVT for screen %d (attempt %d/%d)\n",
+                    i,
+                    attempt + 1,
+                    ENTER_VT_RETRIES);
+
+            if (xf86Screens[i]->EnterVT(xf86Screens[i])) {
+                entered = TRUE;
+                break;
+            }
+
+            /*
+             * FatalError() would cause ddxGiveUp() to call LeaveVT()
+             * for a screen whose vtSema is TRUE anyway.  Do that
+             * cleanup explicitly before retrying EnterVT().
+             */
+            xf86Screens[i]->LeaveVT(xf86Screens[i]);
+            xf86Screens[i]->vtSema = FALSE;
+
+            if (attempt + 1 == ENTER_VT_RETRIES)
+                break;
+
+            xf86Msg(X_WARNING,
+                    "EnterVT failed for screen %d; LeaveVT completed, "
+                    "retrying in %d ms (attempt %d/%d)\n",
+                    i,
+                    ENTER_VT_RETRY_USEC / 1000,
+                    attempt + 1,
+                    ENTER_VT_RETRIES);
+
+            /*
+             * Use an absolute CLOCK_MONOTONIC deadline.  Repeated
+             * signals may interrupt clock_nanosleep(), but they
+             * cannot restart the retry interval from zero.
+             */
+            {
+                struct timespec deadline;
+                int ret;
+
+                clock_gettime(CLOCK_MONOTONIC, &deadline);
+
+                deadline.tv_nsec +=
+                    (long) ENTER_VT_RETRY_USEC * 1000L;
+                if (deadline.tv_nsec >= 1000000000L) {
+                    deadline.tv_sec++;
+                    deadline.tv_nsec -= 1000000000L;
+                }
+
+                do {
+                    ret = clock_nanosleep(CLOCK_MONOTONIC,
+                                          TIMER_ABSTIME,
+                                          &deadline,
+                                          NULL);
+                } while (ret == EINTR);
+            }
+        }
+
+        if (!entered)
+            FatalError("EnterVT failed for screen %d after %d attempts\n",
+                       i, ENTER_VT_RETRIES);
     }
+
     for (i = 0; i < xf86NumGPUScreens; i++) {
         xf86GPUScreens[i]->vtSema = TRUE;
         if (!xf86GPUScreens[i]->EnterVT(xf86GPUScreens[i]))

It causes warning "Starting EnterVT for screen %d (attempt %d/%d)" in Xorg logs for every S3 wake-up event even when everything works correctly but other than that, I haven't seen any side-effects and it was able to avoid one Xorg crash already:

[  4259.594] (WW) Starting EnterVT for screen 0 (attempt 1/10)
[  4259.640] (EE) NVIDIA(GPU-0): Failed to acquire modesetting permission.
[  4259.643] (WW) NVIDIA(0): Failed to set the display configuration
[  4259.643] (WW) NVIDIA(0):  - Setting a mode on head 0 failed: Insufficient permissions
[  4259.643] (WW) NVIDIA(0):  - Setting a mode on head 1 failed: Insufficient permissions
[  4259.643] (WW) NVIDIA(0):  - Setting a mode on head 2 failed: Insufficient permissions
[  4259.643] (WW) NVIDIA(0):  - Setting a mode on head 3 failed: Insufficient permissions
[  4259.643] (WW) EnterVT failed for screen 0; LeaveVT completed, retrying in 100 ms (attempt 1/10)
[  4259.744] (WW) Starting EnterVT for screen 0 (attempt 2/10)

and everything worked as expected after that.

If you want to try this patch on Ubuntu/Debian system, following should be good enough:

mkdir test-nvidia-workaround
cd test-nvidia-workaround
apt source xserver-xorg-core
cd xorg-server-*
patch -p1 < .../path-to-above-patch.diff
dch --local +retry-EnterVT
debuild -uc -us
cd ..
sudo dpkg -i ./xserver-xorg-core_*.deb

The machine probe log for the same event follows (this might be worth checking for Nvidia engineers if they cannot reproduce the bug otherwise - other people without access to source code may not benefit from this):

97992824760083 pm_restore_console ENTER pid=98030 comm=systemd-sleep
97992824765602 vt_move_to_console ENTER pid=98030 comm=systemd-sleep target=6 alloc=0
97992824787664 vt_move_to_console EXIT pid=98030 comm=systemd-sleep ret=62
97992824788165 pm_restore_console EXIT  pid=98030 comm=systemd-sleep ret=0
97992832226523 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=38
97992833435636 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=38 ret=-4
97992833444886 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=38
97992838431682 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=38 ret=-4
97992838437359 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=38
97992843434070 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=38 ret=-4
97992843443727 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=38
97992848432747 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=38 ret=-4
97992848440163 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=38
97992853430502 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=38 ret=-4
97992853434559 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=38
97992858430215 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=38 ret=-4
97992858432479 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=38
97992863429943 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=38 ret=-4
97992863432465 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=38
97992868432344 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=38 ret=-4
97992868439819 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=38
97992870356862 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=38 ret=-1
97992870377174 drm_file_alloc pid=2600 comm=Xorg minor=128 type=2
97992870384280 drm_file_alloc EXIT pid=2600 comm=Xorg ret=0xffff8ac3c379b200
97992870393114 drm_file_free pid=2600 comm=Xorg id=46 minor=128 type=2
97992870416891 drm_file_alloc pid=2600 comm=Xorg minor=1 type=0
97992870422593 drm_file_alloc EXIT pid=2600 comm=Xorg ret=0xffff8ac3c379b200
97992870423354 drm_master_open ENTER pid=2600 comm=Xorg
97992870424565 nv_drm_master_set ENTER pid=2600 comm=Xorg
97992870425091 drm_master_open EXIT pid=2600 comm=Xorg ret=0
97992870453124 nvkms_open ENTER pid=2600 comm=Xorg
97992873430514 nvkms_open EXIT pid=2600 comm=Xorg ret=-4
97992873433595 drm_file_free pid=2600 comm=Xorg id=47 minor=1 type=0
97992873436615 drm_master_release ENTER pid=2600 comm=Xorg
97992873481482 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=39
97992873891161 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=39 ret=-1
97992873905064 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97992874127625 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=-1
97992874163019 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=39
97992874164200 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=39 ret=-1
97992974280724 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=38
97992974285310 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=38 ret=-1
97992974304288 drm_file_alloc pid=2600 comm=Xorg minor=128 type=2
97992974315882 drm_file_alloc EXIT pid=2600 comm=Xorg ret=0xffff8ac3ca2f1e00
97992974323615 drm_file_free pid=2600 comm=Xorg id=48 minor=128 type=2
97992974350651 drm_file_alloc pid=2600 comm=Xorg minor=1 type=0
97992974354490 drm_file_alloc EXIT pid=2600 comm=Xorg ret=0xffff8ac3ca2f1e00
97992974355144 drm_master_open ENTER pid=2600 comm=Xorg
97992974358682 nv_drm_master_set ENTER pid=2600 comm=Xorg
97992974359227 drm_master_open EXIT pid=2600 comm=Xorg ret=0
97992974386575 nvkms_open ENTER pid=2600 comm=Xorg
97992974387865 nvkms_open EXIT pid=2600 comm=Xorg ret=0
97992974389083 drm_ioctl_kernel ENTER pid=2600 comm=Xorg func=nv_drm_grant_permission_ioctl flags=0x2
97992974390181 nv_drm_grant_permission_ioctl ENTER pid=2600 comm=Xorg fd=24 dpyId=0x0 type=3
97992998138524 nv_drm_grant_permission_ioctl EXIT pid=2600 comm=Xorg type=3 ret=0
97992998144066 drm_ioctl_kernel EXIT  pid=2600 comm=Xorg func=nv_drm_grant_permission_ioctl flags=0x2 ret=0
97992998154818 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=41
97992998156773 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=41 ret=0
97992998159027 nvkms_close ENTER pid=2600 comm=Xorg
97992998162703 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=4
97992998164865 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=4 ret=0
97992998165440 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=4
97992998165740 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=4 ret=0
97992998166316 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=4
97992998166608 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=4 ret=0
97992998171711 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=6
97992998342929 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=6 ret=0
97992998369547 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=6
97992998375585 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=6 ret=0
97992998384366 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=6
97992998692299 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=6 ret=0
97992998703668 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=6
97992998710720 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=6 ret=0
97992998720365 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=6
97992999029612 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=6 ret=0
97992999039215 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=6
97992999906748 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=6 ret=0
97992999920514 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=6
97993003110678 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=6 ret=0
97993003123893 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=46
97993003125193 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=46 ret=0
97993009237008 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993013220876 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993013243839 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993016205244 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993016224006 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993019144627 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993019156797 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993021671228 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993021679961 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993025420473 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993025434150 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993028880707 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993028892986 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993032275668 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993032287217 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993035719240 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993035730787 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993038831350 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993038843246 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993041649113 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993041658705 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993047227594 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993047240853 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993050524589 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993050534733 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993054111691 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993054121410 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993057441004 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993057448658 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993060477967 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993060540843 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993063981895 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993064044101 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993067119945 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993067182106 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993070119590 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993070173139 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993073386185 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993073437950 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993076966423 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993077018559 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993080268438 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993080317429 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993083930832 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993083940798 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993087190392 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993088238168 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=9
97993123126571 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=9 ret=0
97993123137465 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=59
97993123322013 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=59 ret=0
97993123329704 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=12
97993123345041 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=12 ret=0
97993123501796 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=18
97993127097261 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=18 ret=0
97993127142836 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=17
97993127203277 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=17 ret=0
97993127305407 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993127461667 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993127739887 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=50
97993127744034 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=50 ret=0
97993180584019 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993180831887 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993180856617 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=50
97993180857632 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=50 ret=0
97993180882351 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=50
97993180882717 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=50 ret=0
97993183225184 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993183470663 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993183495908 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=50
97993183496713 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=50 ret=0
97993183516488 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993183675419 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993183683161 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=50
97993183683608 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=50 ret=0
97993183697179 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993183860237 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993183868263 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=50
97993183868729 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=50 ret=0
97993183881861 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993184046020 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993184053902 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=50
97993184054331 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=50 ret=0
97993184067818 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=50
97993184068965 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=50 ret=0
97993184073797 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=50
97993184074199 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=50 ret=0
97993184119545 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993184272790 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993184281437 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=50
97993184282513 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=50 ret=0
97993186925314 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186929589 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186931060 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186931419 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186931836 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186932101 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186932490 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186932743 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186933178 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186933426 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186933801 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186934050 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186934429 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186934681 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186935059 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186935375 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186935754 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186936079 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186936484 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186936732 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186937117 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186937360 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186937739 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186937997 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186938372 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186938717 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186939091 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186939392 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186939764 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186940009 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186940383 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186940639 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186941012 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186941254 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186941624 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186941873 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186942246 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186942493 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186942864 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186943128 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186943497 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186943741 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186944111 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186944373 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186944748 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186944992 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186945357 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186945712 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186946085 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186946424 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186946795 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186947033 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186947402 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186947651 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186948022 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186948320 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186948693 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186949015 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186949381 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186949617 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186950014 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186950312 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186950682 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186951023 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186951388 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186951628 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186952009 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186952250 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186952619 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186952866 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186953237 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186953564 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186953935 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186954240 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186954612 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186954852 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186955232 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186955474 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186955842 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186956082 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186956450 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186956694 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186957062 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186957525 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993186957907 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=32
97993186958157 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=32 ret=0
97993187391214 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993187641344 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993187927903 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993188091223 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993189461715 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993189637852 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993190572748 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993190744755 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993190758017 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=50
97993190758960 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=50 ret=0
97993195511970 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993195753468 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993195833922 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=50
97993195835884 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=50 ret=0
97993196435358 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993196672380 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993198216037 nvkms_ioctl ENTER pid=4790 tid=4790 comm=chrome inner=21
97993198224396 nvkms_unlocked_ioctl EXIT pid=4790 tid=4790 comm=chrome inner=21 ret=0
97993198597970 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993198816572 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993198853032 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=50
97993198857534 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=50 ret=0
97993198877558 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=50
97993198880114 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=50 ret=0
97993199059212 nvkms_open ENTER pid=2600 comm=Xorg
97993199076813 nvkms_open EXIT pid=2600 comm=Xorg ret=0
97993199079196 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=19
97993199087338 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=19 ret=0
97993199110725 nvkms_ioctl ENTER pid=4790 tid=4790 comm=chrome inner=20
97993199115050 nvkms_unlocked_ioctl EXIT pid=4790 tid=4790 comm=chrome inner=20 ret=0
97993199116500 nvkms_close ENTER pid=4790 comm=chrome
97993199286081 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993199530037 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993200231551 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993200486457 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993200599248 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=50
97993200602354 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=50 ret=0
97993200819242 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=10
97993200822451 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=10 ret=0
97993200824180 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=11
97993200827569 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=11 ret=0
97993200829676 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=10
97993200834300 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=10 ret=0
97993200856378 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993201049401 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993202239664 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993202499655 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993205154837 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993205413602 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993208905877 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993209148859 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993209366816 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993209548503 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993210421256 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993210664885 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993211965604 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993212216393 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993221076513 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993221348471 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993222959208 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993223211703 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993224316188 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993224577243 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993236785618 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993237035041 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993277346575 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993277640398 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
97993300679017 nvkms_ioctl ENTER pid=2600 tid=2600 comm=Xorg inner=14
97993300937288 nvkms_unlocked_ioctl EXIT pid=2600 tid=2600 comm=Xorg inner=14 ret=0
98003047526961 nvkms_open ENTER pid=2600 comm=Xorg
98003047543386 nvkms_open EXIT pid=2600 comm=Xorg ret=0
98003047549470 nvkms_close ENTER pid=2600 comm=Xorg

[mikkorantalainen]

I think Nvidia driver should catch EINTR and retry internally but it doesn’t, which causes Xorg to crash when Nvidia driver internally gets EINTR. If you run lowlatency kernel like I do, the probability for getting EINTR is increased but this will randomly crash generic kernel, too, because EINTR can happen for practically any system call when the whole kernel happens to be running suitable processes in parallel.

The above workaround modifies Xorg to blindly call LeaveVT() and retry if EnterVT() fails. It blindly assumes that EnterVT() failed because of Nvidia internal bug.

This workaround might be generic enough to be usable for other hardware, too, but I couldn’t find official documentation about the assumed system state after EnterVT() fails. Historically Xorg has immediately aborted after that so I would assume the code paths haven't been tested in many drivers... If this were POSIX-like API, then EnterVT() should have no side-effects if it fails but in case of Nvidia drivers, this is not true and repeating EnterVT() will keep failing if LeaveVT() isn’t called in between. (Yes, I tried that first.)

Nvidia should still fix their drivers but at least above workaround can avoid crashing the whole desktop!