ov_core::TrackKLT class

KLT tracking of features.

This is the implementation of a KLT visual frontend for tracking sparse features. We can track either monocular cameras across time (temporally) along with stereo cameras which we also track across time (temporally) but track from left to right to find the stereo correspondence information also. This uses the calcOpticalFlowPyrLK OpenCV function to do the KLT tracking.

Base classes

class TrackBase
Visual feature tracking base class.

Constructors, destructors, conversion operators

TrackKLT(std::unordered_map<size_t, std::shared_ptr<CamBase>> cameras, int numfeats, int numaruco, bool binocular, HistogramMethod histmethod, int fast_threshold, int gridx, int gridy, int minpxdist) explicit
Public constructor with configuration variables.

Public functions

void feed_new_camera(const CameraData& message) virtual
Process a new image.

Protected functions

void feed_monocular(const CameraData& message, size_t msg_id)
Process a new monocular image.
void feed_stereo(const CameraData& message, size_t msg_id_left, size_t msg_id_right)
Process new stereo pair of images.
void perform_detection_monocular(const std::vector<cv::Mat>& img0pyr, const cv::Mat& mask0, std::vector<cv::KeyPoint>& pts0, std::vector<size_t>& ids0)
Detects new features in the current image.
void perform_detection_stereo(const std::vector<cv::Mat>& img0pyr, const std::vector<cv::Mat>& img1pyr, const cv::Mat& mask0, const cv::Mat& mask1, size_t cam_id_left, size_t cam_id_right, std::vector<cv::KeyPoint>& pts0, std::vector<cv::KeyPoint>& pts1, std::vector<size_t>& ids0, std::vector<size_t>& ids1)
Detects new features in the current stereo pair.
void perform_matching(const std::vector<cv::Mat>& img0pyr, const std::vector<cv::Mat>& img1pyr, std::vector<cv::KeyPoint>& pts0, std::vector<cv::KeyPoint>& pts1, size_t id0, size_t id1, std::vector<uchar>& mask_out)
KLT track between two images, and do RANSAC afterwards.

Function documentation

ov_core::TrackKLT::TrackKLT(std::unordered_map<size_t, std::shared_ptr<CamBase>> cameras, int numfeats, int numaruco, bool binocular, HistogramMethod histmethod, int fast_threshold, int gridx, int gridy, int minpxdist) explicit

Public constructor with configuration variables.

Parameters
cameras camera calibration object which has all camera intrinsics in it
numfeats number of features we want want to track (i.e. track 200 points from frame to frame)
numaruco the max id of the arucotags, so we ensure that we start our non-auroc features above this value
binocular if we should do binocular feature tracking or stereo if there are multiple cameras
histmethod what type of histogram pre-processing should be done (histogram eq?)
fast_threshold FAST detection threshold
gridx size of grid in the x-direction / u-direction
gridy size of grid in the y-direction / v-direction
minpxdist features need to be at least this number pixels away from each other

void ov_core::TrackKLT::feed_new_camera(const CameraData& message) virtual

Process a new image.

Parameters
message Contains our timestamp, images, and camera ids

void ov_core::TrackKLT::feed_monocular(const CameraData& message, size_t msg_id) protected

Process a new monocular image.

Parameters
message Contains our timestamp, images, and camera ids
msg_id the camera index in message data vector

void ov_core::TrackKLT::feed_stereo(const CameraData& message, size_t msg_id_left, size_t msg_id_right) protected

Process new stereo pair of images.

Parameters
message Contains our timestamp, images, and camera ids
msg_id_left first image index in message data vector
msg_id_right second image index in message data vector

void ov_core::TrackKLT::perform_detection_monocular(const std::vector<cv::Mat>& img0pyr, const cv::Mat& mask0, std::vector<cv::KeyPoint>& pts0, std::vector<size_t>& ids0) protected

Detects new features in the current image.

Parameters
img0pyr image we will detect features on (first level of pyramid)
mask0 mask which has what ROI we do not want features in
pts0 vector of currently extracted keypoints in this image
ids0 vector of feature ids for each currently extracted keypoint

Given an image and its currently extracted features, this will try to add new features if needed. Will try to always have the "max_features" being tracked through KLT at each timestep. Passed images should already be grayscaled.

void ov_core::TrackKLT::perform_detection_stereo(const std::vector<cv::Mat>& img0pyr, const std::vector<cv::Mat>& img1pyr, const cv::Mat& mask0, const cv::Mat& mask1, size_t cam_id_left, size_t cam_id_right, std::vector<cv::KeyPoint>& pts0, std::vector<cv::KeyPoint>& pts1, std::vector<size_t>& ids0, std::vector<size_t>& ids1) protected

Detects new features in the current stereo pair.

Parameters
img0pyr left image we will detect features on (first level of pyramid)
img1pyr right image we will detect features on (first level of pyramid)
mask0 mask which has what ROI we do not want features in
mask1 mask which has what ROI we do not want features in
cam_id_left first camera sensor id
cam_id_right second camera sensor id
pts0 left vector of currently extracted keypoints
pts1 right vector of currently extracted keypoints
ids0 left vector of feature ids for each currently extracted keypoint
ids1 right vector of feature ids for each currently extracted keypoint

This does the same logic as the perform_detection_monocular() function, but we also enforce stereo contraints. So we detect features in the left image, and then KLT track them onto the right image. If we have valid tracks, then we have both the keypoint on the left and its matching point in the right image. Will try to always have the "max_features" being tracked through KLT at each timestep.

void ov_core::TrackKLT::perform_matching(const std::vector<cv::Mat>& img0pyr, const std::vector<cv::Mat>& img1pyr, std::vector<cv::KeyPoint>& pts0, std::vector<cv::KeyPoint>& pts1, size_t id0, size_t id1, std::vector<uchar>& mask_out) protected

KLT track between two images, and do RANSAC afterwards.

Parameters
img0pyr starting image pyramid
img1pyr image pyramid we want to track too
pts0 starting points
pts1 points we have tracked
id0 id of the first camera
id1 id of the second camera
mask_out what points had valid tracks

This will track features from the first image into the second image. The two point vectors will be of equal size, but the mask_out variable will specify which points are good or bad. If the second vector is non-empty, it will be used as an initial guess of where the keypoints are in the second image.