How to use pixel coordinates to look up texture in shader

Hi,

I’ve tried to find an example or thread in the forum that I could use pixel coordinate to look up texture but I couldn’t.

Like “sampler2DRect” and “texture2DRect” in GLSL 120, what is equivalent in GLSL 150? and how I should set my texture in .cpp?

I’ve tried declare

gl::enable(GL_TEXTURE_RECTANGLE_ARB);

and set my texture with

gl::Texture::Format()
.target(GL_TEXTURE_RECTANGLE_ARB)
.minFilter(GL_NEAREST)
.magFilter(GL_NEAREST)
.internalFormat(GL_RGB32F_ARB)

and tried to pass the texture to my shader with “Sampler2DRect”, but it seems that I still need to look up it with normalized coordinates.
Any insight would be appreciated!

Thanks!

hi,

I vaguely remember last time when I tried rectangle texture I had to use GL_TEXTURE_RECTANGLE_EXT instead of the ARB one and I was on a PC with radeon gpu. Then when sampling your sampler2DRect texture you need to use texture2DRect function.

Also if you are a pc then probably a compute buffer would do better than a rectangle texture nowadays.

Hi @seph

Thanks for a fast reply. I’ve just tried with GL_TEXTURE_RECTANGLE_EXT but seems like it’s deprecated or something. Cinder doesn’t recognize it. Also it doesn’t seem like GLSL 150 have texture2DRect function. Perhaps I’ve missed something. Thanks again though.

Only reason I would like to try to look up a texture with pixel coordinates is that I’ve been trying to experiment something which is sort of like very hard to describe in words… but if I simplify it to core concept that I need would be,

  • a. draw a pixel on the screen, let’s say (x, y) and store it into a frame buffer. (* this pixel must be drawn in vertex shader… so this pixel’s position needs to be store in vertex-array and pass to the shader)
  • b. I need to access the exact pixel in fragment shader by using texture coordinates

I’ve tried it with normalized coordinate but I couldn’t access exact point, seemed like slightly off.
So I assume this problem could be a floating point precision of normalized texture coordinates (which I also stored it in vertex-array-object and identical with vertex’s position) and that’s why I’ll like to try it with screen based pixel coordinate with rectangular texture. However I’m not sure this would be a right approach to be honest.

FYI, I’ve tried to use texelFetch() instead of texture(). But I couldn’t make it run and couldn’t find relative examples in Cinder samples.

Any insight on “other directions” or “how to on texelfetch” or “how to on rectangular texture” would be really grateful!

I believe you just call texture in glsl 150 and if the sampler type is sampler2DRect the uvs are treated as pixels rather than normalised values.

Hi @lithium

Thanks for the reply. I’ve tried it but no luck though.
When I try with sampler2drect, it just gives me a black screen with both normalized coords or pixel coords.
Would you mind taking a look my code?

#include "cinder/app/App.h"
#include "cinder/app/RendererGl.h"
#include "cinder/gl/gl.h"

using namespace ci;
using namespace ci::app;
using namespace std;

class pixelcoordsApp : public App {
public:
    static void prepareSettings( Settings *settings );
    void setup() override;
    void update() override;
    void draw() override;
    
    gl::FboRef m_fbo;
    gl::GlslProgRef m_shdr;
    
    gl::TextureRef m_tex;
    
    string vert =
    CI_GLSL( 150,
            in vec4	ciPosition;
            uniform mat4 ciModelViewProjection;
            void main(){
                gl_Position = ciModelViewProjection * ciPosition;
            });
    
    string frag =
    CI_GLSL( 150,
            uniform sampler2DRect img_tex;
//            uniform sampler2D img_tex;
            uniform vec2 res;
            out vec4 fragColor;
            
            void main(){
                vec2 coords = gl_FragCoord.xy;
//                coords /= res;
                
                vec3 c = texture(img_tex, coords).rgb;
                
                fragColor = vec4(c,1);
            });
};

void pixelcoordsApp::setup()
{
    gl::enable(GL_TEXTURE_RECTANGLE_ARB);
    
    m_shdr =
    gl::GlslProg::create(gl::GlslProg::Format()
                         .vertex(vert)
                         .fragment(frag)
                         );
    
    gl::Fbo::Format fbo_format;
    fbo_format
    .setColorTextureFormat(gl::Texture::Format()
                           .target(GL_TEXTURE_RECTANGLE_ARB)
                           //                           .target(GL_TEXTURE_2D)
                           .minFilter(GL_NEAREST)
                           .magFilter(GL_NEAREST)
                           .internalFormat(GL_RGB32F_ARB)
                           //                           .internalFormat(GL_RGBA32F)
                           .wrap(GL_CLAMP_TO_EDGE)
                           );
    
    m_fbo = gl::Fbo::create( getWindowWidth(), getWindowWidth(), fbo_format.colorTexture() );
    
    m_tex = gl::Texture::create(loadImage(loadAsset("img.png")));
}

void pixelcoordsApp::update()
{
    gl::ScopedFramebuffer fbo_scope(m_fbo);
    {
        gl::clear(Color(.0f,.0f,.0f));
        gl::ScopedViewport viewportScope( ivec2( 0 ), m_fbo->getSize() );
        gl::setMatricesWindow( m_fbo->getSize() );
        
        gl::ScopedGlslProg shdr_scope(m_shdr);
        {
            gl::ScopedTextureBind tex0(m_tex, 0);
            m_shdr->uniform("img_tex", 0);
            m_shdr->uniform("res", vec2(getWindowWidth(),getWindowHeight()));
            
            gl::drawSolidRect(Rectf(0, 0, m_fbo->getWidth(), m_fbo->getHeight()));
        }
        
    }
}

void pixelcoordsApp::draw()
{
    gl::setMatricesWindow( getWindowSize() );
    gl::clear(Color(0, 0, 0));
    
    gl::draw(m_fbo->getColorTexture(), toPoints( m_fbo->getColorTexture()->getBounds() ) );
}

void pixelcoordsApp::prepareSettings( Settings* settings ){
    settings->setWindowSize( 850, 850 );
}

CINDER_APP( pixelcoordsApp, RendererGl, pixelcoordsApp::prepareSettings )

*Attached the image that I use in this code

I just leave here in case its what you need. I don’t think you can ‘draw pixels’ in vertex. You only have one shader per vertex, and when you pass a varying to the next stage it interpotales the value like TextCoord.

In vertex, just pass through with the texture coordinates:

#version 150

uniform mat4	ciModelViewProjection;
in vec4			ciPosition;
in vec2			ciTexCoord0;
out vec2		TexCoord0;

void main(void) {
	gl_Position = ciModelViewProjection * ciPosition;
	TexCoord0 = ciTexCoord0;
}

In the Fragment you mus pass the dimentions of your texture as a uniform, the lookup texture function range is [0,1] so:

#version 150
out vec4		oColor;
in vec4			inColor;
in vec2			TexCoord0;
uniform vec2       uTextureSize;
uniform sampler2D	uTex0; //your texture

void main(void) {
   vec2 position = vec2(200,200);//you define
   vec2 uv = vec2( position.x / uTextureSize.x , position.y/uTextureSize.y);
   //vec4 color = texture( uTex0, TexCoord0);//This is what you would normaly do
   vec4 color = texture( uTex0, uv); //retrieves the color at x,y
   oColor = color;
}

In the CPU:

mFboWidth	= 1920;
mFboHeight = 1080;
mBatch = ci::gl::Batch::create(ci::geom::Plane().size(ci::vec2(mFboWidth, mFboHeight)).axes(ci::vec3(-1, 0, 0), ci::vec3(0, 1, 0)).normal(ci::vec3(0, 0, 1)).origin( ci::vec3( mFboWidth * 0.5, mFboHeight * 0.5, 0) ), mGlsl );
auto lambert = ci::gl::ShaderDef().color();


ci::gl::Fbo::Format format;
format.setSamples( 4 ); // uncomment this to enable 4x antialiasing
mFbo= ci::gl::Fbo::create(mFboWidth, mFboHeight, format.colorTexture());

Render to FBO

ci::gl::ScopedFramebuffer fbScp(mFbo);
// clear out the FBO with blue
ci::gl::clear(ci::Color(0.15, 0.5f, 0.1f));
ci::gl::setMatricesWindow(mFbo->getSize());	
// setup the viewport to match the dimensions of the FBO
ci::gl::ScopedViewport scpVp(ci::ivec2(0), mFbo->getSize());

mTexture->bind(mTexture->getId());
int tex = mTexture->getId();
mGlsl->uniform("uTex0", tex);
mGlsl->uniform("uTextureSize", mTexture->getSize());
mBatch->draw();

This has worked for me, may have a typo but let me know.

Just a couple of tiny little issues. If you change these lines, you’ll be greeted with dali in all his overrated glory :wink:

m_fbo = gl::Fbo::create( getWindowWidth(), getWindowWidth(), fbo_format );
m_tex = gl::Texture::create(loadImage(loadAsset("img.jpg")), gl::Texture::Format().target(GL_TEXTURE_RECTANGLE));
1 Like

Thaaaaaaaaaaaaaaaaaanks @lithium!!! It works like a charm! though I can’t agree with your last statement :stuck_out_tongue:

If you don’t mind I have couple questions,

  1. I remember I saw allocating FBO with only .colorTexture() in somewhere in Cinder samples and it seems like working fine in this case without changing it as you suggested. As far as I understand, if I don’t need depth, stencil, scissor extra stages then allocating with .colorTexture() would be more efficient? or it should be still allocated with all of them?
  2. What’s differences between ‘GL_TEXTURE_RECTANGLE’ and ‘GL_TEXTURE_RECTANGLE_ARB’? Is the one with ARB is just older one than the one without?

Thanks again for your help! Really appreciate! :slight_smile:

  1. When you call fbo_format.colorTexture() you’re overriding all the setup you did previously and assigning a default gl::Texture::Format to the Fbo; all the settings from your call to fbo_format.setColorTextureFormat are nuked.

  2. As a general rule i don’t use anything with a _ARB or _EXT suffix if I can help it. I’m not sure it really matters in practice but it’s a matter of habit. Someone who knows more about the arb’s deprecation / extension model can correct me here.

Hi @xumo,

Thanks for your reply in details. Yes, you’re correct on I can’t technically draw something on vertex shader, sometimes, it’s really hard to explain something in very simple sentence if the something has more complex process behind it heh… Sorry for any confusion. I’ve been trying to implement this paper and one of the trick which this paper introduces is draw vertices(particles) into buffer and color it with it’s coordinates and access it in different stage with another texture which actually has vertices’ position values in it. So I can pull the current vertex’s position and use that position value as a coordinate to lookup the first texture and lookup around each particle’s neighbor’s pixels to figure the collisions. This is kinda why that I said draw point in vertex shader which doesn’t make sense in graphics pipeline. I have a following fragment shader in my actual code. :slight_smile:

Oh, just check the definition of function in Fbo.h, like you said, it overwrites with default color texture.

Format& colorTexture( const Texture::Format &textureFormat = getDefaultColorTextureFormat( true ) ) { mColorTexture = true; mColorTextureFormat = textureFormat; return *this; }

I’m not sure where I saw the weird example that I mentioned above but it seems apparently wrong and yours is correct. Shame on me :sweat_smile: Thanks again for your answer!!

About EXT and ARB: these stand for “extension” and “Architecture Review Board”. Individuals and companies can propose changes and additions (extensions) to the OpenGL standard. These are then reviewed by the ARB. If accepted, they become part of the standard.

Not all proposed additions are accepted, but driver vendors can decide to support them anyway. Extensions like these often have the _EXT suffix, or _ARB while being reviewed. Accepted additions subsequently drop the suffix.

I would suggest to first try using the version without suffix. GL_TEXTURE_RECTANGLE has been around long enough to be fully accepted into the standard.

If your driver does not define the one without suffix, you can try using the EXT or ARB one. Most drivers define all variants, just to avoid any problems with older software. You’ll notice that they all have the same value.

-Paul

Hi @paul.houx

Thanks for your answer in depth! Glad to know why opengl has various extensions in data type and what they really means :slight_smile: Thanks again!