# SLI multicasting with multisampled FBOs

**URL:** <https://discourse.libcinder.org/t/sli-multicasting-with-multisampled-fbos/1123>\
**Category:** Using Cinder\
**Created:** [April 4, 2018, 6:21pm UTC](https://discourse.libcinder.org/t/sli-multicasting-with-multisampled-fbos/1123 "2018-04-04T18:21:05Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![totalgee](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.libcinder.org/totalgee/32/274_2.png) [@totalgee](https://discourse.libcinder.org/u/totalgee)\
**Post date:** [April 4, 2018, 6:21pm UTC](https://discourse.libcinder.org/t/sli-multicasting-with-multisampled-fbos/1123/1 "2018-04-04T18:21:05Z")

</div>

Hi there, this is probably a very “niche” question. I am trying to use the NVIDIA SLI multicast extension ([GL\_NV\_gpu\_multicast](https://www.khronos.org/registry/OpenGL/extensions/NV/NV_gpu_multicast.txt)) with Cinder, to get better performance in VR. It seems to be working fine, and I’m definitely seeing performance improvements (so far around 1.7x frame rate improvement in my particular scenario, with two GTX 1080s). However, I’ve run into a mysterious and frustrating thing…

I’m rendering (for VR) to two Cinder FBOs, normally with 4 multisamples. When SLI multicast is available, I render the left eye into GPU 0’s left FBO, and simultaneously the right eye into GPU 1’s “left” FBO. Then, when the rendering is done (after making a _single_ set of draw calls to draw both eyes!), I need to blit the GPU 1’s “left” FBO’s contents back to GPU 0 as its “right”. This is required prior to submission of the two eye textures to OpenVR (or even just to draw on screen, say). This transfer is done with a call to glMulticastBlitFramebufferNV() (which is like the regular glBlitFramebuffer call, but allows you to explicitly transfer between GPUs).

What I’m seeing is that the right-eye (after transferring back to GPU 0) is not multisampled. It is all aliased, like the resolve didn’t happen properly (or more like rendering was done without multisampling, since you can’t visualize a MS buffer). I have called fbo-\>resolveTextures() and prior to that bindFramebuffer(), which ensures it’s marked as “dirty” so it does the resolve.

Strangely, my exact same code works glitch-free if I use 8 or 16 samples in the FBOs, and also works if I turn on CSAA sampling. But it fails (the right eye is aliased) when I have 2 or 4 samples. Of course, it also works fine (both eyes aliased) when no multisampling is enabled.

I know it’s a long shot, but has anyone _any_ thoughts on what might be wrong? I’ve tried all kinds of combinations of things, but keep seeing the same result. Maybe it’s even a driver bug, but somehow I doubt that.

Thanks,  
Glen.

---

<div class="post-metadata">

**Author:** ![totalgee](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.libcinder.org/totalgee/32/274_2.png) [@totalgee](https://discourse.libcinder.org/u/totalgee)\
**Post date:** [April 10, 2018, 6:24pm UTC](https://discourse.libcinder.org/t/sli-multicasting-with-multisampled-fbos/1123/2 "2018-04-10T18:24:43Z")

</div>

Just for the record, I finally managed to sort this out (after first writing a pure OpenGL program to try to repro it, then finding the pure OGL program _didn’t_ exhibit the bug! 😉 ). Ah, so it was time to start stepping into the details of what Cinder is actually doing.

To do the multi-GPU stuff I want, I kind of need to “wrangle” a Cinder FBO. But Cinder kind of protects you from the underlying details, keeping both multisampled and resolved versions in a single Fbo instance (normally a good thing!). But I need to transfer the _resolved_ version from one GPU to the other.

So in the end it wasn’t really complicated, but I was confused by the various “intricacies”, such as that simply calling `Fbo::bindFramebuffer()` also marks the framebuffer as dirty (needing multisample resolve). It does that even if you just bind it for reading with `GL_READ_FRAMEBUFFER`. (Devs: maybe that should be changed, only flagging it as dirty for `GL_DRAW_FRAMEBUFFER` or `GL_FRAMEBUFFER` bindings?)

Also, I needed more granular control of the multisample vs resolved “sub” framebuffers within one Cinder Fbo object, in particular to explicitly bind the resolved version – luckily there is an `Fbo::getResolveId()` that lets you do everything directly, with raw OGL calls. For awhile I was also confused because I thought `Fbo::getId()` was returning that (as opposed to `Fbo::getMultisampleId()`), but of course in the multisample case they both return the same value. (Yes, this is mentioned in the header…I just missed it the first time! 😉 )

Anyhow – it’s very cool, now everything works, double GPU action now, even when multisampling is active.

Thanks,  
Glen.

---

<div class="post-metadata">

**Author:** ![paul.houx](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.libcinder.org/paul.houx/32/31_2.png) [@paul.houx](https://discourse.libcinder.org/u/paul.houx)\
**Post date:** [April 10, 2018, 8:36pm UTC](https://discourse.libcinder.org/t/sli-multicasting-with-multisampled-fbos/1123/3 "2018-04-10T20:36:03Z")

</div>

Thanks for sharing your insights, Glen!

---

<div class="post-metadata">

**Author:** ![rich.e](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.libcinder.org/rich.e/32/1025_2.png) [@rich.e](https://discourse.libcinder.org/u/rich.e)\
**Post date:** [April 11, 2018, 7:05pm UTC](https://discourse.libcinder.org/t/sli-multicasting-with-multisampled-fbos/1123/4 "2018-04-11T19:05:32Z")

</div>

Hi there,

I wanted to mention that there’s been a bit of thought around how `ci::gl::Fbo` could be made more flexible and intuitive, as you can see on [this list](https://github.com/simongeilfus/Cinder/issues/2). I’m not sure if his work in progress addresses your needs, but when the time comes I’m sure it’d be great to get feedback or suggestions on how to make things like what you’re doing easier.

cheers,  
Rich

---

<div class="post-metadata">

**Author:** ![totalgee](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.libcinder.org/totalgee/32/274_2.png) [@totalgee](https://discourse.libcinder.org/u/totalgee)\
**Post date:** [April 12, 2018, 8:08am UTC](https://discourse.libcinder.org/t/sli-multicasting-with-multisampled-fbos/1123/5 "2018-04-12T08:08:37Z")

</div>

Thanks for pointing me to that. Just in case it’s useful to him, I added a comment there.

---

<div class="post-metadata">

**Author:** ![gaoyu](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.libcinder.org/gaoyu/32/142_2.png) [@gaoyu](https://discourse.libcinder.org/u/gaoyu)\
**Post date:** [February 22, 2019, 6:40am UTC](https://discourse.libcinder.org/t/sli-multicasting-with-multisampled-fbos/1123/6 "2019-02-22T06:40:51Z")

</div>

Can you share a little code for how to transfer from one GPU to the other?  
Thank you

---

<div class="post-metadata">

**Author:** ![gaoyu](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.libcinder.org/gaoyu/32/142_2.png) [@gaoyu](https://discourse.libcinder.org/u/gaoyu)\
**Post date:** [February 22, 2019, 6:51am UTC](https://discourse.libcinder.org/t/sli-multicasting-with-multisampled-fbos/1123/7 "2019-02-22T06:51:22Z")

</div>

hi totalgee  
How to select one of gpus to do tasks?

---

<div class="post-metadata">

**Author:** ![totalgee](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.libcinder.org/totalgee/32/274_2.png) [@totalgee](https://discourse.libcinder.org/u/totalgee)\
**Post date:** [February 23, 2019, 12:40am UTC](https://discourse.libcinder.org/t/sli-multicasting-with-multisampled-fbos/1123/8 "2019-02-23T00:40:50Z")

</div>

Hi there. Yes, it’s a bit complicated. I really suggest you look at the description of the NVIDIA OpenGL extension:  
[https://www.khronos.org/registry/OpenGL/extensions/NV/NV\_gpu\_multicast.txt](https://www.khronos.org/registry/OpenGL/extensions/NV/NV_gpu_multicast.txt)

But in quick summary (from what I recall – it’s been awhile), to select a GPU (or several) to perform subsequent rendering commands, you would use:

```auto
void glRenderGpuMaskNV(bitfield mask);

```

With (e.g.) `0x1` as mask for the first GPU, `0x2` for the second, `0x3` for both.

And to transfer data (a framebuffer at the end of rendering) from one GPU to the other (e.g to get the right eye’s rendering back to the left-eye GPU to submit to a VR compositor), you would use:

```auto
void glMulticastBlitFramebufferNV(uint srcGpu, uint dstGpu,
                                    int srcX0, int srcY0, int srcX1, int srcY1,
                                    int dstX0, int dstY0, int dstX1, int dstY1,
                                    bitfield mask, enum filter);

```

Before doing the blit, however, you’d likely need to wait for the other GPU to finish, so you use:

```auto
void glMulticastWaitSyncNV(uint signalGpu, bitfield waitGpuMask);

```

You also need to `WaitSync` after doing the blit, before submitting to the VR compositor (or using the transferred buffer in some other way).

I also found I needed to have the environment variable `GL_NV_GPU_MULTICAST` set to `1` before doing any OpenGL stuff, for the extension to even show up as available when queried.

The extension (link shown above) has some example code, but it can be a bit tricky to get it working properly. One potential point of confusion is that some methods take a GPU mask, others take the index (0-based) of the GPU.

Obviously, you don’t want to render the same thing on both GPUs. So for example in the stereo rendering case, you want the same geometry and uniforms to be sent to both GPUs, but the viewing/projection matrices to be different. You do this by setting the uniform buffer that contains the camera transform (at least) to have a “per GPU” storage (`GL_PER_GPU_STORAGE_BIT_NV` or’ed into the flags for `glBufferStorage()`). And then to set different data to each “version” of the UBO, you’d use the extension method:

```auto
void glMulticastBufferSubDataNV(
        bitfield gpuMask, uint buffer,
        intptr offset, sizeiptr size,
        const void *data);

```

There is more you can do with it, this is just one use case. Again, look at the extension documentation.

Hope that helps a bit,  
Glen.