First-Estimate Jacobian Estimators
When developing an extended Kalman filter (EKF), one needs to linearize the nonlinear motion and measurement models about some linearization point. This linearization is one of the sources of error causing inaccuracies in the estimates (in addition to, for exmaple, model errors and measurement noise). Let us consider the following linearized error-state visual-inertial system:
where the state contains the inertial navigation state and a single environmental feature (noting that we do not include biases to simplify the derivations):
Note that we use the left quaternion error state (see [Indirect Kalman Filter for 3D Attitude Estimation]  for details). For simplicity we assume that the camera and IMU frame have an identity transform. We can compute the measurement Jacobian of a given feature based on the perspective projection camera model at the k-th timestep as follows:
The state-transition (or system Jacobian) matrix from timestep k-1 to k as (see [IMU Propagation Derivations] for more details):
where is the true acceleration at time , is computed using the gyroscope angular velocity measurements, and is gravity in the global frame of reference. During propagation one would need to solve these integrals using either analytical or numerical integration, while we here are interested in how the state evolves in order to examine its observability.
The observability matrix of this linearized system is defined by:
where is the measurement Jacobian at timestep k and is the compounded state transition (system Jacobian) matrix from timestep 0 to k. For a given block row of this matrix, we have:
We now verify the following nullspace which corresponds to the global yaw about gravity and global IMU and feature positions:
It is not difficult to verify that . Thus this is a nullspace of the system, which clearly shows that there are the four unobserable directions (global yaw and position) of visual-inertial systems.
The main idea of First-Estimate Jacobains (FEJ) approaches is to ensure that the state transition and Jacobian matrices are evaluated at correct linearization points such that the above observability analysis will hold true. For those interested in the technical details please take a look at:  and . Let us first consider a small thought experiment of how the standard Kalman filter computes its state transition matrix. From a timestep zero to one it will use the current estimates from state zero forward in time. At the next timestep after it updates the state with measurements from other sensors, it will compute the state transition with the updated values to evolve the state to timestep two. This causes a miss-match in the "continuity" of the state transition matrix which when multiply sequentially should represent the evolution from time zero to time two.
As shown above, we wish to compute the state transition matrix from the k-1 timestep given all k-1 measurements up until the current propagated timestep k+1 given all k measurements. The right side of the above equation is how one would normally perform this in a Kalman filter framework. corresponds to propagating from the k-1 update time to the k timestep. One would then normally perform the k'th update to the state and then propagate from this updated state to the newest timestep (i.e. the state transition matrix). This clearly is different then if one was to compute the state transition from time k-1 to the k+1 timestep as the second state transition is evaluated at the different linearization point! To fix this, we can change the linearization point we evaluate these at:
We also need to ensure that our measurement Jacobians match the linearization point of the state transition matrix. Thus they also need to be evaluated at the linearization point instead of the that one would normally use. This gives way to the name FEJ since we will evaluate the Jacobians at the same linearization point to ensure that the nullspace remains valid. For example if we evaluated the Jacobian with a different at each timestep then the nullspace would not hold past the first time instance.