OK, so I managed to get your code up and running. I had to fix a few things (removed
using namespace from the headers, removed the
CinderExtensions.h include, upgraded to toolset v120 and fixed several warnings and compiler errors).
Most of the code stayed the same, with the exception of your
VecExt class, which I presume is defined in the missing
CinderExtensions.h. I replaced the code that uses
VecExt with the GLM equivalent. For instance:
vec3 vSrc = vecExt.transformVec( targetToSource, vec3( x, y, 1 ) );
vec3 vSrc = targetToSource * vec3( x, y, 1 );
I was able to drag&drop an image file, set it to Bilinear projection and select one of the four edges. I could move the edge by dragging it without any problems when the "Zoom" slider was all the way to the left. When zoomed in, I was no longer able to select an edge or vertex, indicating that your coordinate system conversion was incorrect.
I tracked the error and it appears to be the use of
glm::affineInverse() where you should really use
glm::inverse() instead (so my initial guess was spot on, haha).
Also, just be careful when transforming positions using a
mat4: make sure the w-coordinate is set to 1. For instance:
vec3 transformPoint = vec3( mViewToShape * vec4( pos, 0, 1 ) );