The approach is focusing on the detection of road lane markings under the use of common computer vision algorithms in combination with the OpenCV Library (Open Computer Vision Library) (v.3.1) and ROS (Robot Operating System) (version: kinetic).
Several computer vision techniques are used to separate desired features from the input image with the purpose to provide a robust and fast analysing algorithm for lane keeping assistants.
Inverse Perspective Mapping (IPM) is applied to easily achieve 2D-to-3D (I <-> W) coordinate-transformation to simplify pixel to real distance mapping. Large gradient peaks are particularly searched by the standard Sobel-Kernel which is a very commonly used tool for edge-detection.
To improve quality and safety of lane extraction from the input set of data the IPM image is additionally divided into multiple of rectangular bins (number of bins – NOB) of predefined height where several of lines can be fitted to achieve better quality in lane clustering. The data points of well isolated lane markings are performed under the use of second-degree polynomial regression and least squares method.
Lastly, Kálmáns prediction and estimation filter (Kalman-Filter) is used for position tracking and correction to keep the test vessel safely inside of the lane, to finally support work on fully autonomous and self-driving vehicles.
Image Preprocessing – Histogram Equalisation/ Stretching
Histogram equalisation is simply understood as a method of improving the image-contrast where the resulting image appears more detailed than the original one. Histogram examples of over- and underexposed grey-scale images show that these ones seem to carry less information in it as images with equalised or stretched histograms. To undertake histogram equalisation, a cumulative distribution function (cdf) needs to be calculated with the probability distribution of the intensity values of a given image. Depending on environmental conditions histogram equalisation can response in very over- and/or underexposed results , that’s because histrogram stretching is applied which can provide a more or less equalised(stretched) histogram depending on the strength of stretching.
The resulting image(right image below) with the stretched histogram responds in an improved level of contrast where more relevant features of lane markings can be detected.
Inverse Perspective Mapping
Inverse perspective mapping(IPM) is applied to remove the perspective effect of the camera from the input image to obtain an undistorted top-view where it is assumed that lane markings will appear as quasi parallel lines to each other to describe global real-world coordinates with pixel positions on an image.
A 4 by 4 point correspondence is used to achieve the desired top-view, as it is illustrated in the middle one of the three pictures below.
A 3×3 homography matrix H can simply be determined through a linear system of 8 equations. The generated top-view has the advantage of a direct mapping of physical real-world units to pixel units without any further need of calculating additional world- to image coordinate transformations.
However, the accuracy is still hardly limited to the chosen resolution of the image and the physical units which are mapped to an individual imager element. In this application the accuracy is a compromise between quality, spoken in resolution and time of processing. Therefore, the resolution is chosen in such a way that 6cm of real-world units will be mapped to a single pixel.
The approach combines two different ways of searching for gradient peaks in the camera image to increase the final outcome of actual features of lane markings which need to be detected securely under various of scenarios. Edge detecting filter-matrices, such as the Sobel-Kernel, are tuned to identify specifically one type of a gradient orientation. Therefore, an additional simplified routine of searching gradient peaks is applied to the original and perspective distorted image to be in the position to not just filter for one specific orientation but for multiple ones. The combined methods of searching gradient peaks is significantly improving the finding of lane markings where also under strongly bended and curvy scenarios a feature-rich set can be provided.
Sobel-Filtered Gradient Search
The filtering starts with the use of the Sobel-Operator in x direction
which is highly responding to vertical aligned edges in an image, the result of filtering can be seen in the image below.
This type of a filter makes it to a superb candidate for the finding of a specific oriented lane marking. Further, the highest magnitudes of found features getting collected as part of the final set of features.
Search for Gradients
Next to the orientation of a lane marking, its width also appears differently in the input image by considering straight and curved lane segments. The image gets searched through several rows of pixels where the width of a lane marking can differ largely due to the corresponding scenario.
To generalize searching, a system of perspectively aligned vectors is used which is determining the minimum and maximum width a lane marking is allowed to have in both straight and bended cases.
Sum of Gaussians – SOG
Sum of Gaussians is used to determine an approximation for a probability density function on base of discrete values to find out about the most dense regions of filtered image features.
The search of dense regions is basically undertaken by simply counting detected pixels in column-wise order of potential edges aligned in horizontal direction. The so provided discrete distribution of column values is further used to determine the number and the place of potential lanes which have previously been detected. A sum of multiple of gaussian functions(left image) is lastly revealing the previously mentioned most dominant and dense parts of detected edge pixels of the current image(right image). Further, the maxima get simply determined with the help of the first and second derivative of the sum of gaussians.
Separating found features into an individual set of lanes can be quite difficult by considering the set of features as a whole. The existence of noise and false detected lane features as well as bended and curvy lane segments makes it even harder to get a clear boundary of which features actually belong to a specific lane. Therefore, the image gets divided into a set of a predefined number of rectangular bins where found features will be interpreted with smaller linear line segments as a subpart of an entire lane. This, specifically helps with the work of bended and curved lane segments to securely divide the set of features into individual lanes, the image below is illustrating the approach of dividing the image in multiple of bins.
The smaller linear line segments will be formed by the use of a RANSAC algorithm where false segments can still be isolated with the knowledge about the average slope of all line segments. The initial starting points for the RANSAC algorithms were determined by the previously calculated sum of gaussians (SOG). The line segments of the RANSAC algorithms need to be examined by the existence of intersecting points of neighbored lines where an entire lane will be identified by the finding of overlapping line segments. In previous approaches it has always been difficult to clearly identify one individual lane especially by considering curved lane segments and the partial lost of lane marking information. The found parts and patches of existing lane markings were too close together to clearly identify one individual lane under the use of distance measurements from one line segments to each other.
Therefore, a new approach of identifying individual lane markings(left or right lane marking) has been developed. A lane marking still gets determined with the help of simple distance measurements. Hereby, the midpoint of a line segment is used to calculate the distance to a potential intersection point with another line segment. Generally, a small distance indicates that the corresponding line segments are segments of the same lane marking, accordingly, a large distance indicates that the corresponding line segments are not part of the same lane. To finally form an individual lane a DBSCAN algorithm is used to form clusters of single line segments.
Higher-Order Polynomial Regression
Polynomial regression helps to further refine the found clustering and to describe the lane model where the continuous flow of road lanes can suitably be interpreted with second-degree polynomial regression and least squares method.
The lane and vehicle corresponding coefficients of
will be determined by the use of least squares method where a polynomial equation is fitted to the points of an individual cluster of line segments.
Tracking of Polynomial Coefficients: Kalman-Filter
Kalman filter equations are used to get an estimation of second-degree polynomial coefficients to track the state of the underlying detection.
The desired centre line gets simply calculated by frequently updated left and right lane marking information. The actual detected lane markings are also interpreted as second-degree polynomial equations where the midline represents the average distance between them.
The state-vector x (image below) of the Kalman filter will be prepared with the lane to vehicle corresponding coefficients of, kappa – curvature, sin(psi) – yaw-angle and y_0 – transverse direction of the vehicle which is resulting in a 3-dimensional Kalman filter:
Furthermore, the transition matrix A is prepared with the derivatives of the previously introduced second-degree polynomial equation where x is set to v*Δt. The Kalman filter is very powerful to predict the most likely state of the future of a filtered system component. Therefore, not just the change of the system can be calculated to every Δt from one update to another but the filter is also in the position to predict upcoming states for the future.