Canvas jitter/flicker when resizing with multithreaded render loop

support
1

I have an F# windowing system built on top of GLFW that allows easy creation and management of multiple windows. The rendering is done via Skia (specifically SkiaSharp). Recently, I moved the rendering for each window from being synchronous (i.e., occurring on the main/UI thread along with the window management) to a unique thread for each window. With this setup, each window gets a unique, standalone thread that completely handles the rendering.

However, moving the rendering to a thread has introduced some jitter in the shapes drawn and also some flickering of the canvas (sometimes going black). Below are some videos of the behavior.

(Note: the graininess of the GIFs is due to the video to GIF conversion process to make everything small enough to upload.)

Single threaded window management and rendering, all on the same thread:

Single-threaded rendering - no jitter

Multithreaded environment, with window management on the main/UI thread and rendering in a single, unique thread:

Multi-threaded rendering - jitter

The code is effectively identical between the two situations outside of the threaded behavior.

Even though the rendering is occurring on another thread, when the refresh window or framebuffer resize callbacks are called, I make synchronous calls to regenerate the canvas (recreate the Skia surface with the new framebuffer size, redraw the canvas, and then swap buffers) before allowing the callbacks to finish. Even with that, this jitter behavior still persists.

Curiously enough, if I add a sleep of like 1s inside the render loop after recreating the surface and swapping buffers, this seems to get rid of this. It almost seems like there’s some multithreaded issue with the context/rendering functions that allows a resize to occur while a callback is being executed and/or before a buffer swap begins or completes. Or it seems possible that the swap buffers call is returning too early, before it has finished with the buffer swap.

Here’s my callback setup:

    member private this.DefaultWindowRefreshCallback (_window: GLFWWindow) =
        if _debugFlag then printfn "%s refreshed" _title
        let width, height = getFrameBufferSize _window
        _framebufferWidth <- width
        _framebufferHeight <- height
        this.RegenerateCanvas()

    member private this.DefaultFramebufferSizeCallback (GLFWWindow(_window), width: int, height: int) =
        if _debugFlag then printfn "%s framebuffer resized: (width, height) = (%d, %d)" _title width height
        _framebufferWidth <- uint16 width
        _framebufferHeight <- uint16 height
        this.RegenerateCanvas()

The RegenerateCanvas call makes a synchronous call in both situations, whether the actual rendering occurs on another thread or not. In the threaded cases, a message is sent and a reply is awaited, where the reply is sent after the buffers are swapped. Here’s the code that does the framebuffer resize and drawing in both situations. GenerateCanvas is called first and then Draw.

    member this.GenerateCanvas bundle =
        if _debugFlag then printfn "Generating new canvas..."
        //printfn "generating canvas %A" renderState.Canvases

        this._bundle <- bundle

        // Dispose of existing canvas
        if this._surface <> null then this._surface.Dispose()
        if this._backendRenderTarget <> null then this._backendRenderTarget.Dispose()

        // Create new canvas
        this._backendRenderTarget <- new GRBackendRenderTarget(int this._bundle.FramebufferWidth,
                                                               int this._bundle.FramebufferHeight,
                                                               this._bundle.Samples,
                                                               this._bundle.StencilBits,
                                                               this._framebufferInfo)
        this._surface <- SKSurface.Create(this._grContext,
                                          this._backendRenderTarget,
                                          GRSurfaceOrigin.BottomLeft,
                                          SKColorType.Rgba8888)
        this._canvas <- this._surface.Canvas

    member this.Draw () =
        _drawFunction this._canvas
        this._canvas.Flush()
        swapBuffers _window

The bundle contains the framebuffer width and height. The _drawFunction draws the shape at static positions (i.e., does not change between render frames).

Polling versus waiting for events doesn’t seem to affect this. Changing the swap interval between 1 and 0 seems to have some effect, the latter reducing the frequency of the jitter but not removing it.

I am thinking that there may be a possible issue with the timing relationship between when resizes on the framebuffer are allowed, the relevant callbacks, and the swap buffer function.

Thank you for any help on this! I have read many posts here in the GLFW forum and also posts in Stack Overflow, but I have not been able to figure this out.

2

I’m beginning to be somewhat convinced that this is indeed some race condition issue between the resize events and the end-to-end surface recreation and buffer swap process.

If I alter my framebuffer resize callback to disable window resizing while in the callback, this issue seems to go away. (Of course, this does have the effect of making the maximize icon flicker.)

    member private this.DefaultFramebufferSizeCallback (GLFWWindow(_window), width: int, height: int) =
        setWindowAttribute _window (Resizable false)
        if _debugFlag then printfn "%s framebuffer resized: (width, height) = (%d, %d)" _title width height
        _framebufferWidth <- uint16 width
        _framebufferHeight <- uint16 height
        this.RegenerateCanvas()
        setWindowAttribute _window (Resizable true)
3

Here’s a video of the case described directly above, where the rendering is happening in a separate thread but the window is set to not be resizable while the framebuffer resize callback is being executed.

Unless anyone has some ideas, this seems like a bug somewhere in GLFW or things it calls into, so I might file an issue on the GLFW repository. I’m hoping there is an easy fix either to what I’m doing or to GLFW. For now, this workaround is workable but is certainly not ideal given that the maximize button now flashes for slow resizes.

Multi-threaded rendering blocking resizes during resize - no jitter

4

Getting threaded rendering correct is a difficult task and I would not immediately assume there is a bug in GLFW or the OpenGL implementation if your code works single threaded but has issues when the rendering is threaded.

You might want to check if using glfwGetFramebufferSize in the render threads rather than using the size callback might help if the underlying problem is due to the size not being correctly updated between threads before the rendering - it likely probably won’t help but it’s simple to check.

Out of interest what OS / driver / GPU are you using?

EDIT: Quick thought - you might want to ensure that when a resize event has occurred you only call glfwPollEvents after you have completed the next render with that size. This would synchronize the rendering and event loop during resizing and could eliminate resizing during rendering.

5

I appreciate not jumping to conclusions, and I originally assumed I was doing something wrong. But after these investigations, I’m currently not sure of that. (I actually hope that’s the case because it would be easily fixable.) I’m in F#, so it’s actually not hard at all to do multithreaded code. In the multithreaded case, the main thread and the render thread do everything inside their respective threads in a serialized manner. The two threads share data only via messaging, so there is no shared state aside from the window handle.

The RegenerateCanvas method occurs on the main thread. In the single-threaded case, it recreates the canvas synchronously. In the multi-threaded case, the RegenerateCanvas method, still occurring in the callback, sends a message to the render thread with the new framebuffer size and waits (i.e., blocks inside the callback) for a reply. The render thread receives the message to recreate the canvas with the new framebuffer size, recreates the Skia surface (and thus canvas), redraws, swaps buffers (that’s all in the GenerateCanvas method above), and only then does it send the reply back to the blocking call inside RegenerateCanvas, which then allows the framebuffer size callback to complete.

A few further notes:

  • I am using the glfwWaitEvents call in the main thread and am not polling. This was a conscious choice to support low CPU GUIs but then split out rendering to another thread to support asynchronous rendering for animation use cases without needing to poll for events.
  • My understanding was that the framebuffer size callback occurs on the main thread, which should prevent any call to wait events. Is that correct? Again, all the calls on the main thread are serialized explicitly, aside from the callbacks. It is a current assumption that wait events and the callback could not be running at the same time.
  • I can’t call glfwGetFramebufferSize inside the render thread since it is on a different thread than the main thread.
  • Events should not be firing or at least not handled while I am inside any callback, correct?

What makes me think it is a bug is that the issue goes away when I prevent resizing while inside the callback.

I am on Windows 11 with a desktop Intel i7 and Nvidia GeForce RTX 3060 Ti.

6

The framebuffer size callback does occur on the main thread. On Windows it will likely occur multiple times during resizing inside a call to glfwWaitEvents or glfwPollEvents whilst the user holds down the mouse button whilst resizing as the event loop becomes blocked during this time. This is a Win32 API issue, see issues #185, #561, #1231 and the PR #1426. The behaviour of your own single threaded example seems different to what I am normally used to, for example if you try the GLFW boing example you should observe that resizing the window clears the resized area and it doesn’t get redrawn until resizing stops.

Correct - my fault as I forgot this isn’t thread safe.

Events can be fired by the OS, but are not being processed by the app whilst you are inside your callback - so potentially if your callback is taking too long to render the window size might have changed before rendering is complete.

One robust solution to the the above problem I have seen used is to create an undecorated window and handle resizing yourself by drawing a resize handle and resizing the window when this is dragged. This will ensure that no resize events occur when your application is still drawing.

Another approach (I have not tested this) might be that when you receive the resize callback you use glfwSetWindowSizeLimits to clamp the window size to the current size, then send the message to redraw the canvas but return immediately as glfwSetWindowSizeLimits requires events to be processed to work. Once rendering of the canvas is complete send a message to the main thread to unclamp the window size by calling glfwSetWindowSizeLimits with GLFW_DONT_CARE as the min and max width and height parameters.

Thanks - this helps with some of the explanation above.

7

Thank you for keeping on taking a look at this with me!

I know little about OpenGL, but it looks like that example doesn’t actually try to display anything during the framebuffer size callback. In their callback, they are just calling reshape, which appears to just resize the viewport, and display doesn’t get called until the next loop after the events have been polled (and supposedly handled). If you add the following lines at the end of reshape:

display();
glfwSwapBuffers(window);

then that gets the redraw behavior during the resizes. (I tested this on Windows with Visual Studio 2022.)

boing resize

The first workaround seems a bit daunting in having to reimplement resizing. I can try the second one (haven’t yet), but it’s somewhat similar to the workaround I have currently that prevents resizes while in the framebuffer size callback.

I’m still not seeing anything that explains the differing behavior between the single-thread and multi-thread cases. In both cases, my calls within the framebuffer size callback (DefaultFramebufferSizeCallback) to regenerate the canvas (RegenerateCanvas) are synchronous. The only thing that changes is that the canvas is created and managed on another thread and glfwMakeContextCurrent, glfwSwapInterval, and glfwSwapBuffers are called on that same render thread, while glfwWaitEvents and the framebuffer size callback still run on the main thread.

I also haven’t seen relation to the length of execution time of my framebuffer size callback. It’s buried in the original post, but if I explicitly and intentionally lengthen the time of the callback inside the render thread, the issue goes away. The fact that it shows up more the faster things are and it doesn’t happen on every frame seems to imply to me that there’s a race condition somewhere.

8

Apologies, I’m not able to spend much time helping so I missed the fact your single threaded case also ran the rendering from the callback (this isn’t usual in the realtime applications I normally work with).

Nor I, but GLFW does not have any complicated threading code - glfwMakeContextCurrent , glfwSwapInterval, and glfwSwapBuffers all call underlying OpenGL functionality, and NVIDIA’s OpenGL drivers are normally fairly robust.

If I get time over the weekend I might be able to work up a simple example of something similar in C++ to check if I can get the same behaviour.

1 Like
9

No worries! I do appreciate you taking a look at this! Thanks for the discussion so far. I’m so close to being done with the whole windowing system and getting to the stuff of the GUI framework (controls, styles, transitions, layout, etc.). :crossed_fingers: After I get passed this hurdle, I’ll mainly need to just circle back and implement remaining window properties and window/input callbacks.

The reason for the synchronous rendering in this case is only due to the nature of the framebuffer resize. I otherwise intend on running the renderer fully asynchronous from any other callback, as most of them get dispatched to a window specific event listener that can be consumed by the broader application (basically that window even listener and processor offloads the event from the main thread as fast as possible).

My goal here is to make a GUI framework that, for the most part, defaults to GUI windows that wait for events and have static renders. But I want the GUI system to support animations within a “static” GUI, and I want to support full animation windows that display some 2D graphics, such as what Processing, Pure Data, or vvvv do. So that’s the main reason for my current architecture.

The repository is private right now, because this is all in work, but I am willing to add helpful debuggers as contributors to the repo to get some eyes on it. It is F#, but the code should be highly readable for the most part, at least particular to the GLFW stuff.

10

I don’t have time to help with debugging your project specifically, but hopefully I will have time to write a simple threaded renderer with resizing redraw support as an example / test.

11

Oh, no worries. I should have clarified, but I was more just mentioning that I’m willing to share the code if it helps.

Thank you for taking some time to look at it and come up with a C++ test and example. I don’t know C++ well enough (can read it to gather some information) to easily come up with an example. It will be interesting to see if the behavior reproduces or not. I currently can’t see a way out, other than the workaround I currently have, so having a hint where the bug or misstep is at (my code, GLFW, platform API, or driver) would help.

12

I have found out that, at least from what I can currently tell, this issue doesn’t occur on every Windows machine. It occurs on two Windows 11 machines that have NVIDIA graphics cards, but from what I can see in my initial testing, I don’t see it occur on a Windows 10 machine with only Intel Iris Xe. Of course, the issue is intermittent, so it can be shifty to determine if it’s actually gone. Although it does normally show up somewhat readily.

So it seems possible that this is a SkiaSharp or Skia issue, for example see [BUG] [WinUI] Scale delay is observed during window resize · Issue #2341 · mono/SkiaSharp (github.com) or it is a graphics driver issue.

There are issues with SkiaSharp on Linux right now, see [BUG] Exit code 139 (segmentation fault) when calling GRGlInterface.Create on Ubuntu and Linux Mint · Issue #2350 · mono/SkiaSharp (github.com), so I can’t test there at the moment. And I don’t currently have a macOS machine to test there.

13

I hate to resurrect this thread, but I am still dealing with trying to debug this issue, as it still occurs. And now, it actually occurs in even the most simple, single-threaded cases. It still does not ever occur when on integrated Intel graphics, and it always occurs when using NVIDIA graphics.

See videos here regarding the differing behavior between the graphics with a framebuffer resize callback implemented: [BUG] Drawing is warped during window resize on NVIDIA graphics card · Issue #2832 · mono/SkiaSharp (github.com)

Here’s the code for a reproducing case. It is in F#, but it’s directly using SkiaSharp bindings to Skia and Silk.NET.GLFW bindings to GLFW, which are both major projects and not using my own bindings. Also, the calls should be close enough in name to the C++ cases to be understandable.

open FSharp.NativeInterop
open Silk.NET.GLFW
//open Silk.NET.OpenGL
open SkiaSharp

#nowarn "9"

let initialWidth, initialHeight = 500, 500
let mutable framebufferWidth, framebufferHeight = initialWidth, initialHeight
let mutable displayWidth = initialWidth
let mutable displayHeight = initialWidth
let mutable viewportWidth = initialWidth
let mutable viewportHeight = initialHeight

let glfw = Glfw.GetApi()
glfw.Init() |> printfn "Initialized?: %A"

// Uncomment these window hints if on macOS
//glfw.WindowHint(WindowHintInt.ContextVersionMajor, 3)
//glfw.WindowHint(WindowHintInt.ContextVersionMinor, 3)
//glfw.WindowHint(WindowHintBool.OpenGLForwardCompat, true)
//glfw.WindowHint(WindowHintOpenGlProfile.OpenGlProfile, OpenGlProfile.Core)

glfw.WindowHint(WindowHintInt.Samples, 0)
glfw.WindowHint(WindowHintInt.StencilBits, 1)
glfw.WindowHint(WindowHintBool.DoubleBuffer, true)
glfw.WindowHint(WindowHintBool.Focused, false)
glfw.WindowHint(WindowHintBool.Maximized, false)
glfw.WindowHint(WindowHintBool.Visible, true)

let window = glfw.CreateWindow(initialWidth, initialHeight, "Test Window", NativePtr.ofNativeInt 0n, NativePtr.ofNativeInt 0n)
glfw.MakeContextCurrent(window)
glfw.SwapInterval(1)

let grGlInterface = GRGlInterface.Create(fun name -> glfw.GetProcAddress name)

if not (grGlInterface.Validate()) then
    raise (System.Exception("Invalid GRGlInterface"))

let grContext = GRContext.CreateGl(grGlInterface)
//let gl = GL.GetApi(fun name ->
//    printfn "GetAPI name: %A" name
//    glfw.GetProcAddress name)
let grGlFramebufferInfo = new GRGlFramebufferInfo(0u, SKColorType.Rgba8888.ToGlSizedFormat()) // 0x8058
grContext.ResetContext()
//gl.Viewport(0, 0, uint32 framebufferWidth, uint32 framebufferHeight)
let mutable grBackendRenderTarget = new GRBackendRenderTarget(initialWidth, initialHeight, 1, 0, grGlFramebufferInfo)
let mutable surface = SKSurface.Create(grContext, grBackendRenderTarget, GRSurfaceOrigin.BottomLeft, SKColorType.Rgba8888)
let mutable canvas = surface.Canvas

let reshape(window, width, height) =
    framebufferWidth <- width
    framebufferHeight <- height
    //gl.Viewport(0, 0, uint32 width, uint32 height)

    if surface <> null then surface.Dispose()
    if grBackendRenderTarget <> null then grBackendRenderTarget.Dispose()
    
    grBackendRenderTarget <- new GRBackendRenderTarget(width, height, 1, 0, grGlFramebufferInfo)
    surface <- SKSurface.Create(grContext, grBackendRenderTarget, GRSurfaceOrigin.BottomLeft, SKColorType.Rgba8888)
    canvas <- surface.Canvas

    canvas.Clear(SKColors.LightBlue)
    use black = new SKPaint(Color = SKColors.Black)
    canvas.DrawCircle(float32(initialWidth)/2.0f, float32(initialHeight)/2.0f, 100.0f, black)
    canvas.Flush()

    glfw.SwapBuffers(window)

let draw window =
    //gl.Viewport(0, 0, uint32 framebufferWidth, uint32 framebufferHeight)

    canvas.Clear(SKColors.LightBlue)
    use black = new SKPaint(Color = SKColors.Black)
    canvas.DrawCircle(float32(initialWidth)/2.0f, float32(initialHeight)/2.0f, 100.0f, black)
    canvas.Flush()

    glfw.SwapBuffers(window)

let resizeCallbackFun window width height = reshape(window, width, height)
glfw.SetFramebufferSizeCallback(window, resizeCallbackFun) |> ignore

while glfw.WindowShouldClose window <> true do
    glfw.WaitEvents()
    draw window

glfw.DestroyWindow window
glfw.Terminate()
  • On integrated Intel graphics, with no framebuffer resize callback set, the scene is never stretched. It’s just the case that a black region beyond the original drawing area is created.
  • On NVIDI graphics, with no framebuffer resize callback set, the scene is stretched when the window is resized to be bigger and is compressed when the window is resized to be smaller.

See videos here that demonstrate this differing behavior between the two graphics: Resizing a window with GLFW (Silk.NET) and Skia (SkiaSharp) with no resize callback - Album on Imgur

This behavior leads me to believe that something is allowing the window resize to occur prior to the framebuffer resize callback being occurred. This would explain why in the NVIDIA case the jitter is always such that the shape is stretched or compressed and then immediately set back to the correct size when the framebuffer resize callback is called and corrects the image. Since the integrated Intel graphics case never warps the image during resize, this would explain why that case always allows the framebuffer resize callback to set the correct size.