ov_core::TrackBase class

Visual feature tracking base class.

Reference

This is the base class for all our visual trackers. The goal here is to provide a common interface so all underlying trackers can simply hide away all the complexities. We have something called the "feature database" which has all the tracking information inside of it. The user can ask this database for features which can then be used in an MSCKF or batch-based setting. The feature tracks store both the raw (distorted) and undistorted/normalized values. Right now we just support two camera models, see: undistort_point_brown() and undistort_point_fisheye().

A Note on Multi-Threading Support

There is some support for asynchronous multi-threaded feature tracking of independent cameras. The key assumption during implementation is that the user will not try to track on the same camera in parallel, and instead call on different cameras. For example, if I have two cameras, I can either sequentially call the feed function, or I spin each of these into separate threads and wait for their return. The currid is atomic to allow for multiple threads to access it without issue and ensure that all features have unique id values. We also have mutex for access for the calibration and previous images and tracks (used during visualization). It should be noted that if a thread calls visualization, it might hang or the feed thread might, due to acquiring the mutex for that specific camera id / feed.

This base class also handles most of the heavy lifting with the visualization, but the sub-classes can override this and do their own logic if they want (i.e. the TrackAruco has its own logic for visualization). This visualization needs access to the prior images and their tracks, thus must synchronise in the case of multi-threading. This shouldn't impact performance, but high frequency visualization calls can negatively effect the performance.

Derived classes

class TrackAruco: Tracking of OpenCV Aruoc tags.
class TrackDescriptor: Descriptor-based visual tracking.
class TrackKLT: KLT tracking of features.
class TrackSIM: Simulated tracker for when we already have uv measurements!

Public types

enum HistogramMethod { NONE, HISTOGRAM, CLAHE }: Desired pre-processing image method.

Constructors, destructors, conversion operators

TrackBase(std::unordered_map<size_t, std::shared_ptr<CamBase>> cameras, int numfeats, int numaruco, bool stereo, HistogramMethod histmethod): Public constructor with configuration variables.

Public functions

void feed_new_camera(const CameraData& message) pure virtual: Process a new image.
void display_active(cv::Mat& img_out, int r1, int g1, int b1, int r2, int g2, int b2, std::string overlay = "") virtual: Shows features extracted in the last image.
void display_history(cv::Mat& img_out, int r1, int g1, int b1, int r2, int g2, int b2, std::vector<size_t> highlighted = {}, std::string overlay = "") virtual: Shows a "trail" for each feature (i.e. its history)
auto get_feature_database() -> std::shared_ptr<FeatureDatabase>: Get the feature database with all the track information.
void change_feat_id(size_t id_old, size_t id_new): Changes the ID of an actively tracked feature to another one.
auto get_last_obs() -> std::unordered_map<size_t, std::vector<cv::KeyPoint>>: Getter method for active features in the last frame (observations per camera)
auto get_last_ids() -> std::unordered_map<size_t, std::vector<size_t>>: Getter method for active features in the last frame (ids per camera)
auto get_num_features() -> int: Getter method for number of active features.
void set_num_features(int _num_features): Setter method for number of active features.

Protected variables

std::unordered_map<size_t, std::shared_ptr<CamBase>> camera_calib: Camera object which has all calibration in it.
std::shared_ptr<FeatureDatabase> database: Database with all our current features.
std::map<size_t, bool> camera_fisheye: If we are a fisheye model or not.
int num_features: Number of features we should try to track frame to frame.
bool use_stereo: If we should use binocular tracking or stereo tracking for multi-camera.
HistogramMethod histogram_method: What histogram equalization method we should pre-process images with?
std::vector<std::mutex> mtx_feeds: Mutexs for our last set of image storage (img_last, pts_last, and ids_last)
std::mutex mtx_last_vars: Mutex for editing the *_last variables.
std::map<size_t, cv::Mat> img_last: Last set of images (use map so all trackers render in the same order)
std::map<size_t, cv::Mat> img_mask_last: Last set of images (use map so all trackers render in the same order)
std::unordered_map<size_t, std::vector<cv::KeyPoint>> pts_last: Last set of tracked points.
std::unordered_map<size_t, std::vector<size_t>> ids_last: Set of IDs of each current feature in the database.
std::atomic<size_t> currid: Master ID for this tracker (atomic to allow for multi-threading)

Function documentation

ov_core::TrackBase::TrackBase(std::unordered_map<size_t, std::shared_ptr<CamBase>> cameras, int numfeats, int numaruco, bool stereo, HistogramMethod histmethod)

Public constructor with configuration variables.

Parameters
cameras	camera calibration object which has all camera intrinsics in it
numfeats	number of features we want want to track (i.e. track 200 points from frame to frame)
numaruco	the max id of the arucotags, so we ensure that we start our non-auroc features above this value
stereo	if we should do stereo feature tracking or binocular
histmethod	what type of histogram pre-processing should be done (histogram eq?)

void ov_core::TrackBase::feed_new_camera(const CameraData& message) pure virtual

Process a new image.

Parameters
message	Contains our timestamp, images, and camera ids

void ov_core::TrackBase::display_active(cv::Mat& img_out, int r1, int g1, int b1, int r2, int g2, int b2, std::string overlay = "") virtual

Shows features extracted in the last image.

Parameters
img_out	image to which we will overlayed features on
r1	first color to draw in
g1	first color to draw in
b1	first color to draw in
r2	second color to draw in
g2	second color to draw in
b2	second color to draw in
overlay	Text overlay to replace to normal "cam0" in the top left of screen

void ov_core::TrackBase::display_history(cv::Mat& img_out, int r1, int g1, int b1, int r2, int g2, int b2, std::vector<size_t> highlighted = {}, std::string overlay = "") virtual

Shows a "trail" for each feature (i.e. its history)

Parameters
img_out	image to which we will overlayed features on
r1	first color to draw in
g1	first color to draw in
b1	first color to draw in
r2	second color to draw in
g2	second color to draw in
b2	second color to draw in
highlighted	unique ids which we wish to highlight (e.g. slam feats)
overlay	Text overlay to replace to normal "cam0" in the top left of screen

std::shared_ptr<FeatureDatabase> ov_core::TrackBase::get_feature_database()

Get the feature database with all the track information.

Returns	FeatureDatabase pointer that one can query for features

void ov_core::TrackBase::change_feat_id(size_t id_old, size_t id_new)

Changes the ID of an actively tracked feature to another one.

Parameters
id_old	Old id we want to change
id_new	Id we want to change the old id to

This function can be helpfull if you detect a loop-closure with an old frame. One could then change the id of an active feature to match the old feature id!