Issue 7377048: Ceres for bundle adjustment (Closed)
DescriptionThis is a WIP patch which switches libmv from using SSBA to Ceres. Originally written by Keir Mierle, then attempted to be finished by self.
Keir, changes since your work there:
OpenCVReprojectionError:
- Warped constants with T() so jet operations are happy
- Added workaround which skips distortion thing if focal_length is zero. This is needed because of couple of reasons:
* EuclideanBundle uses completely empty intrinsics with all the parameters zero.
* Currently bundling happens with normalized undistorted tracks, so doing distortion here doesn't seem to be logical.
* this is all solvable by altering pipeline a bit, but for now i wanted to make bundling of R/t/X working robust. Adding intrinsics refining support after this seems to be easy.
RotationMatrixPlus:
- Trivial change, AngleAxisToRotationMatrix is in ceres namespace.
- And eeh, mixing col and row major matrices here.. From first glance it's all fine here, but feel a bit paranoid about this chunk of code, would double check.
- Made it operator() _const_. Without const AutodiffParameterization::Plus blamed on passing "this" to plus_functor_ which discards qualifier.
AutodiffParameterization:
- Aha, this is where i really want your help. On the one hand it's pretty much stroghtforward what should happen here from math point of view. However, i could be bullshitting from API usage point of view.
- Replaced RotationMatrixParameterization with PlusFunctor. No idea why parameterization was used at first point. According to docs, AutoDiff expects struct with operator() which will compute z for given x and y. Seems plus functor is what we need here. I've also checked local_parameterization_test where QuaternionPlus functor is passed to AutoDiff. Here it seems really quite the same case, so i just went ahead here and did this.
- x_plus_delta eeeh.. Not so much clear from docs.. local_parameterization_test uses different deltas here (zero, near-zero, away-form-zero). autodiff_test doesn't seem to initialize here anything. This blows my mind. And that's where current culprit is i think.
BundleIntrinsicsLogMessage:
- Moved bunch of if(foo) LG << "bar" into this function. Just to make EuclideanBundleCommonIntrinsics a little bit easier to follow
EuclideanBundleCommonIntrinsics:
- Using RotationMatrixPlus for template argument for AutodiffParameterization. it shall be a functor passed here anyway, not a parameterization. Correct me if i'm wrong, but if we want RotationMatrixParameterization then it'll be a subclass of AutodiffParameterization<RotationMatrixParameterization, 9, 3>. Plus() method would be proper functor there and autodiff shall be indifferent to which plus functor is used, it'll just compute derivatives for any X and Delta (and it wouldn't matter how x+delta is compted for it, right?)
- Made it so functor is a separate variable of class RotationMatrixPlus. Otherwise seems some kind of c++ parsing glitch happens and problem.SetParameterization(&camera->R(0, 0), rotation_parameterization) fails to cast rotation_parameterization to LocalParameterization. Spent a while on this, but only could say it seems to be happening because of constructor not accepting any arguments. In this case SomeClass() is threating in some weirdo way (like it fails to distinguish class construction from function prototype)
-Made it so rotation_parameterization is allocated in heap, not stack. Otherwise residual destructor fails because it tries to free this parameterization, which for sure leads to memory corruption.
- Corrected syntax around residual block initialization.
- Apparently, it is possible that no residuals are filled in. In this case added explicit return, so no problem solving happens in this case. Otherwise ceres will abort with assert deper in Eigen because of trying to work with zero-sized matrices. Shall i fire report to Sameer or it is expected behavior?
- If no intrinsics are being refining, mark the whole block as constant
- Use Jacobi preconditioner type, SCHUR_JACOBI requires cholmod (documentation explicitly says this)
- Same for ITERATIVE_SCHUR. Using DENSE_SCHUR now. Not sure, how sparse dense is. As in, shall we finally add cholmod or not?
- Copying intrinsics shall work fine now
So, current issues.
- Well, bundling doesn't work :) Solver reaches parameter tolerance in like 8 iterations and there's no changes in cost function
- Tried to run code without parameterization (thought it'll be just damn a heck slower). And something weird is going on there. Bundle adjustment claims it successfully bundled things, InternalReprojectionError reports really low reprojection error. But! Solution is just bad -- it's instable and bundles reprojection happening in clip editor shows reprojection errors of tens of pixels (much higher than InternalReprojectionError reported). It could be simply because in difference of how reprojection happens in libmv and blender: without parameterization it's possible that R is not actually only a rotation matrix which leads to different reprojections than after R and t were converted to 4x4 matrix.
I know it is crappy to test BA in such a configuration, but i want to double check on InternalReprojectionError and reprojection in clip editor, because they doesn't seem to be correlating here.
- Also noticed when bundling with intrinsics refirenment crash happens. Bang, no ebery today to solve this. Will check tomorrow.
Patch Set 1 #Patch Set 2 : Updated patch set, details a bit later (not enough space in this little text entry!) #Patch Set 3 : Just another update, details later #
Total comments: 14
Patch Set 4 : Solved most of review notes #
Total comments: 3
Patch Set 5 : Update for scoped_ptr and AutoDiff paremeterizaton comment #
MessagesTotal messages: 13
|
||||||||||||||||||||||||||||
