Real-Time Color Image Fusion and its Key Technologies
Ni Guoqiang£¬Xiao Manjun£¬Qin Qingwang
£¨Department of Optical Engineering£¬School of Information Science and Technology£¬
Beijing Institute of technology£¬Beijing 100081£¬China£©
Image fusion technology has been earliest proposed in the late 1970s of 20th century, in order to via certain process the multi-source collected images for the same object, extracting the information of each channel, outputting the uniformed image in the output side finally, for the further decision-making and estimating assignment. The information synthetic processing technology that full utilizing the complementarily of multi-source date as well as intelligent and high speed arithmetic ability of computer, firstly applied in military region, involved each aspect of scouting, aiming, precisely guiding, target detecting, recognizing and tracing, in the civilian region also involved remote sensing and monitoring of earth resource, weather forecast, traffic guidance ,medical image processing and so on.
Image fusion system has an outstanding advantage in detecting, it could overcome the limitation of each single sensor in geometry, spectrum, spatial discrimination and so on, also using its complementarily to improve image quality, which can save time, reduce cost for the post-treatment, and improve the image synthetically application efficiency. Therefore the image fusion technology developing to the present, many countries universally pay attention to this technology, and obtain obvious benefit from practical application.
Generally£¬on the target identification or identity estimation (property discrimination) layer£¬image fusion can be divided into three types, which are pixel grade fusion, characteristic grade fusion and decision grade fusion. Although multiresolution algorithm has already including partial features of implied characteristic level fusion, generally it still reckoned as pixel level fusion method. At present the dominated pixel level algorithm adopts multiresolution analyzing structure, mainly aims at gray fusion method.
With spring up of the high performance new type image sensor, the requirement of the image fusion technology also increased continuously, people pursuit full utilization and dig the present potential of image sensor, however, emphasize the image fusion should accord with the feature of human vision more and more, indicating as natural mode just simple also adapts to eyes observing, which can obviously improve the recognition performance of image fusion, reduce the fatigue feeling of operator. Due to human eyes is more sensitive to color, its resolution is higher than gray image, also has excellent ability of color constancy and dynamic range compression, especially at present the high speed collecting, processing, transmitting of color image has come true, therefore the color image fusion technology according with human vision characteristics and its real-time achievement has become one of the most important developing directions.
This paper will start from the development of color image fusion algorithm, analyzing several difficulties of key technologies of real-time color image fusion (such as multi-source image characteristic analysis, image registration, post-processing of fused image and so on), and sum up the present real-time color image fusion technology applications and its developing trend.
2 the development of color image fusion algorithm
In gray image, human eyes only could simultaneously distinguish gray scale of lower than 100 different types from black to white, however the human eyes resolution for color could reach several hundreds types even if several thousands types. Based on this human eyes feature, if we can represent the detail information contained in the multi-source channel image gray scale by adopting some kinds of color processing technology£¬it can make people have more abundant knowledge about human eyes toward image detail.
The earliest image gray false color coding fusion method, first proceed the gray fusion for the input multi-source images, then coding the fusion output gray image with multicolor code to output the color image. This method is the post-processing of the image fusion; its essential has none direct relation with image fusion, and therefore it¡¯s not the color image fusion method in practical. Later developed color image fusion method is through some processing, then making the demonstrated information difference of among different source image different gray, directly expressed using color and strengthen display, but in the color showed space, also appearing the color image fusion technology , which adopted the different color space (such as LHS, RGB). LHS color space can greatly realize the human eyes reflected to the tone, luminance and saturation. But the terminal equipment of any image processing procedure should resolve into RGB color space, therefore, achieving the color image fusion in RGB color space directly, no doubt have the advantages of computing simple, speed fast, prone to hardware real-time achievement.
At present prevalent image fusion method mostly based on multi-resolution analytical structure. Initial multi-resolution structure is Laplas pyramid proposed by P.J.Burt and E.H.Adelson, based on this has some deformation and extension, accordingly develop some fusion algorithm.
Later£¬its essential also is multi-resolution wavelet analysis of multi-resolution structure, Hessian and Li introduced into image fusion region, and obtained research widespread. Image color obtained by these fusion methods, sometimes never considered the human visual characteristic, color contrast is too vivid, prone to cause viewer fatigue, unsuitable long time viewing, even if the difference between color image after fusion and real color is large, affect the correct judgment of viewer. Therefore, the researcher more and more emphasis on the color image fusion method according with the human vision characteristic.
Image fusion utilizing visual physiology in the abroad started earlier. R.P.Broussard¡¢S. K.Rogers¡¢M.E.Oxley and G.L.Tarr utilize the pulse-coupled neural net to do characteristic grad image fusion, it is claimed that could obviously improve the recognition rate of target. J.Waldemark also utilizes pulse-coupled neural net to do target recognition research of cruise missile. A.M.Waxman of Lincoln laboratory in MIT utilize antagonism accept field£¨imitate rattlesnake bimodal cell operating mechanism£©proposing the antagonism fusion of low level light night vision and infrared image; A.Toet of TNO manpower factor institute proposed that the color mapping method based on the analysis of common and particular components of source image, just similar to the biological color antagonism mechanism.
Recently, the fusion method theory based on the visual characteristic seldom has great breakthrough, dominating is the above related algorithms, and practilization gradually.
The optic and electronic engineering department of Beijing institute of technology is one of the earliest image fusion technology developing institutes, especially doing large numbers of work in color image fusion technology research based on visual characteristic, improving the Waxman and Toet fusion algorithm, providing the foundation for self-developing the real-time color image fusion system.
3 Key technology analyses
3.1 several color image fusion algorithm based on visual characteristic and its correlation analysis.
As we known human vision system has a center-circle antagonism structure, it can generate different excitement or restraint response to the optical signal, which reach the different position in sensing field. This structure is widespread exists in biological system, especially in the bimodal cell structure of rattlesnake view tectory, also can generate sense to the information input, which derived from visible light and infrared image simultaneously. Waxman just utilized this antagonism structure establishing the typical fusion structure, shown as Fig.1, in which CENTER-ON/SURROUND-OFF structure realized the contrast sensing property, the second stage the processing of infrared enhanced low level light , infrared restraint low level light accord with the image processing mechanism of rattlesnake to the visible light and infrared dual channel.
Fig.2 is the schematic figure of visible light and infrared image fusion method proposed by Toet, above all compute the common and particular part of both separately, utilize the comparison of particular part, imitate antagonism structure mapping into the RGB space, generating color fusion effect.
Fig.1 A kind of low level light and infrared fusion structure proposed by Waxman
Fig.2 Fusion method schematic diagram proposed by Toet
Image fusion algorithm it self is just for the interactive processing of different source image information¡ªthat is adopt some method to effectively contain information implied in each image, the most important foundation in this process is relativity analysis of source image. If the correlation coefficient of source images is 1, which means images are perfectly correlated; if the correlation coefficient is -1, which means images are completely independent, that is including numerous useful complementary information; if the correlation coefficient is 0, image is randomly correlated. Since the source images of perfect correlation mapping into the color space couldn¡¯t obtain plenty color information, therefore it is necessary to choose the source image of small correlation coefficient, then obtained fusion image with plenty color information and well color contrast, thereby achieve the purpose of image fusion. Moreover the image algorithm it self should have good correlation extraction and decorrelation ability, that is effectively abstract the common part of source image, and particular part of each image.
The above two methods both adopt the imitate vision characteristic method in analyzing the source image correlation, due to the vision system has abilities of enhancing image common information, enhancing complementary information and well dynamic range compression, thus color image fusion methods established from this angle have achieved excellent effect (see fig.3&4). Other decorrelation methods, such as image statistical method is still under researching.
Fig.3 Fig.1showed Waxman fusion algorithm effect picture 
£¨which picture a is visible light image, picture b is long wave infrared image,
picture c is color fusion effect£©
Fig. 4 Fig. 2 showed Toet fusion algorithm effect picture
£¨which picture a is visible light image, picture b is infrared image,
picture c is color fusion result£©
Documents [22£¬23]improve method based on the Waxman model, theoretically haven¡¯t exceed the range of Waxman antagonism accept field, done some work in keeping information integrity, contrast enhancement, and so on. However the document  starts from the Retinex experiment of Land, proposing a kind of color image fusion method, and comparing with the Toet method. Fig.5 shows the fusion effect of documents [22£¬24].
3.2 The characteristic research of different source image
Image fusion based on the relationship of sensing device, different sensor based on different physical phenomenon, so that obtained different source images may have great difference.
The information channel of image fusion usually comes from visible light imaging, multi-spectrum low level light night viewing, near infrared imaging, short wave, middle wave and long wave infrared thermal imaging, also including SAR imaging, high spectrum imaging, and so on. Information source has great difference, and therefore the color effect of fusion image has great change space. Color image fusion algorithm should enhance the common information, show the complementary information, remove noise, meanwhile if want to express by using colors , which accord with the human vision and sense habit, it is required familiar with characteristic of different source image its self.
For example, the foremost characteristic of low level light is low single noise ratio. When illuminating value lower than 10-31x£¬noise almost submerges the image, the contrast, luminance and resolution will descend obviously, the low level light image single has great spatial and time correlation, however the image noise has very small spatial and time correlation. Also the signal noise ratio of infrared image is smaller than common visible light, and dynamic range is larger, but easily to generate spatial nonuniformity; the infrared image characteristics of different wavebands have difference. Synthesize the different source image characteristics, improving the contrast, variety of color extent of these complementary characteristic in the fusion image, it could effectively improve the fusion image quality.
£¨a£©Waxman fusion effect of modified method £¨b£©the fusion method based on Land experiment
Fig.5 Show the research group partial fusion algorithm effect
3.3 Real-time image registration technology
For many purposes we need to introduce the real-time image fusion technology, one of the difficulties is the images obtained by different sensors having difference in time, spatial position, angle and so on. Therefore the research of real-time image registration technology is crucial.
In order to obtain the well fusion effect, we must confirm that the pixel points of different source images corresponding to the same time and spatial position, that is to say on one hand the real-time image registration should confirm the time of source images is synchronous, on the other hand, it should offset the distortion caused by lens deformation, rectify the different source images spatial translation, rotation or ratio zooming and other problems caused by different sensors without registration.
It is observed from Fig.6, if different source images without registration processing, its fusion effect will generate ¡°ghosting¡± phenomenon, in the high pr¨¦cised application fields (such as target tracing, precise guiding, target variation detecting and so on) will cause a strong impact on accuracy of post-processing. What¡¯s need explaining is not all the image fusion applied fields will need high precise image registration technology. In the without registration condition of Fig.6(a), compare to the main body part (human image) of near field image, in the background region far from the sensor, its constructional image quality still in the acceptance region. When doing the real-time processing£¬the whole scene changes without intermission, its difficult to find a single mapping method, meanwhile achieves the precise registration for the source image different parallactic scene. In some real-time systems, propose this problem individually, set the interactive mode operating interface for choosing of different application environment.
Abroad already have the quite mature real-time image registration technology, embedding the real-time image fusion system. Domestic also develop large number of relative research. At present in the research process still exists may difficulties, especially the achievement of the image high precise and auto-registration technology in the conditions of image or date class large difference (such as optic and SAR image), wave bands large difference( such as visible light and long wave infrared image) , and so on.
At present the difficulty of real-time image registration technology is mainly derived from the isomerous sensor image registration, high precise( sub-pixel grade, deep sub-pixel grade) image registration, high speed image registration, auto-image registration and the image registration of serious position difference, and so on. But in the research process of corporate with the real-time image fusion technology, we should hold the application environment requirement of image registration accuracy and speed, accordingly obtain the registration result of adapted to the image fusion requirement.
Fig.6 The effect of image registration technology applied in the image fusion
3.4 Post-process of the fused image
The design process of color image fusion algorithm has considered the dynamic range compress, the matching of color and human vision and so on, and the farther process of the direct fused result may obtain the color fusion result more close to the natural feeling. This will help to ease the tiredness of the observer.
Literature applied the color remapping to the direct fusion result in HSV space, and for different scenes (like forest, desert and so on) designed different mapping factors (see Figure 7). This color remapping process will increase the contrast of different objects, maintain the color invariance, and be propitious to the display output, for this matter the subject group also developed relative research (real image reproduction technology research). Figure 8 shows the process to the direct fusion result in Literature by using the core of the optic nerve dynamics ---- annular receptive field principle.
Fig. 7 post-processing of fig.4 fusion result Fig. 8 post-processing of fig.4 fusion result
4. The developments of the real-time color image fusion system
At present, there are two ways to achieve the real-time color image fusion system: one is based on DSP technology; another is based on large scale FPGA technology, namely SoPC (programmable SoC) real-time fusion system.
The research of the real-time fusion system by using DSP has achieved a great success in and abroad. Texas Instrument (TI) in US developed the research of the image fusion system earlier, and had a great success. In 1998, the MIT Lincoln Lab. in US designed a real-time fusion system based on C80 processor, in order to realize the color fusion algorithm of low light level CCD image and infrared image, and obtained a good result; in 2000, MIT added mid-infrared and short-infrared camera on the former system, and developed the multi-sensor real-time night vision fusion system. In China, Beijing Institute of Technology, Nanjing Institute of Technology and some other research office also developed the design and experiment of image fusion system and this provide the support for the real applications.
For the recent few years, the fast development of the large scale programmable devices make it been attended much in real-time fusion system field, recently, both China and the other countries are working on the development by using large scale programmable FPGA to achieve SoPC real-time fusion system, and obtained invigorative achievement.
In the early time, OCTEC and Waterfall Solutions Crop.in Britain cooperatively developed a real-time implementation of image alignment and fusion system on police helicopter, using the Stratix II series FPGA from Alera Crop. As the processor, receiving the infrared and visible image separately, and achieved automatically alignment and fusion, the result was directly displayed on the monitor for the pilot to observe. Now this system has been applied practically, but due to the limitation of the automatic alignment level, it needed to load these two cameras with a distance between them, the alignment technique of this system requires a farther research; color fusion algorithm, camera automatic control technique are the points for the next step.
The real-time color image fusion system developing ability of Equinox Crop.in US is on the top of the world. On SPIE Defense Security Symposium in June 2005, Equinox released the latest research achievement DVP-4000. This system roots in the ¡°night vision infrared and low light level image fusion¡± development plan named by SBIR (Small Business Innovation Research) of the US Army/CECOM (US Army Communications-Electronics Command). DVP-4000 picked the Cyclone series and Stratix series FPGA as the processor, the power consumption is only 1.5W, by using the optimized color fusion algorithm form Equinox Crop., this system well achieved automatic alignment and color fusion of two channel infrared and visible images. This is a big step forward to the commercial applications for real-time image fusion system.
In China, the development of real-time image fusion system that based on FPGA is later, still stay in the laboratory.
Currently, the main problems of the real-time color image fusion system are:
? Low weight, small size, portability;
? Low power consumption;
? Achieve automatic alignment;
? Fast image fusion algorithm, achieve real-time fusion;
? The color fusion image is intuitionistic and natural, good color invariance, easy to display.
Therefore, expedite the research of high performance real-time color image system will have the great meaning to driven the development and applications of relative technology.
In the US IDGA (Institute for Defense and Government Advancement) 4th image fusion annual conference held at the beginning of 2006, there is a series of discussion about the meaning and the developing directions of image fusion technology. We can see that many developed countries like US attach much importance to the research of real-time image fusion system, and has already stepped to miniaturization, practicality and commercial applications; and also deepen the research of image fusion technology to a higher level, such as 3D image fusion based on multi-sensors.
On the synthetical angle of view, an excellent image fusion technology also relies on the well development of high performance image sensor. The trend is to integrate complementary information of different source images, and build up a color image fusion method which matched with human vision; the development of real-time fusion system should step to low power consumption, small size/portable and function integration.
ADDRESS£º4th of jiuxianqiao Road, Beijing, China
POST ADDRESS£ºBEIJING 8511 BOX COEMA