GEO Satellite Image Navigation withCloud Detection using Multispectral

Payload Image Data ?

E. Zaunick ∗ K. Janschek ∗∗ J. Levenhagen ∗∗∗

∗ Institute of Automation, Technische Universitat Dresden,Dresden, Germany (Tel.: +49 351 463 31913, Fax:

+49 351 463 37039, e-mail: edgar.zaunick@tu-dresden.de).∗∗ Institute of Automation, Technische Universitat Dresden,Dresden, Germany (e-mail: klaus.janschek@tu-dresden.de).

∗∗∗ EADS Astrium GmbH,Friedrichshafen, Germany (email: jens.levenhagen@astrium.eads.net)

Abstract: Earth observation from geostationary orbit requires extremely accurate pointingknowledge of the instrument. Due to misalignments, thermal distortion effects and uncertaintieson the attitude and position, it is essential to use image information, to meet the stringentrequirements. We developed a high accuracy attitude estimation system, which uses multispec-tral payload image data. The navigation system evaluates the shift of the image content andreconstructs the rotational motion of the satellite. The estimation results are filtered and fusedwith AOCS data. The geometrical image correction and registration of the satellite images isperformed using the improved knowledge of the line of sight.The reliability of image navigation depends fundamentally on the correct recognition ofcloud areas in the image data. Clouds can cover the earth’s surface and images can becomeincomparable as well as cloud motion can be interpreted as false satellite motion. Thus, acloud detection algorithm has been developed, which is based solely on unregistered and non-calibrated image data. A multispectral analysis (MSA) algorithm has been implemented, whichuses VIS and IR channels for cloud detection during day and night time. The paper presents adescription of cloud detection algorithm, results of sensitivity analysis with respect to groundtexture and lightning conditions and simulation results of the navigation performance undercloud conditions.

Keywords: Image Navigation, Image Motion Compensation, Optical Flow, Motion Estimation,Cloud Detection.

1. INTRODUCTION

Geostationary Earth observation weather satellites acquiremulti-spectral images during operational mode using aline scanner. Satellite rotational and translational motionduring the image acquisition phase cause geometricalimage distortions. Currently this problem is solved by theprinciple of landmark tracking using ground control points(GCP). It uses feature recognition (e.g. edge detection)of shorelines Liu and Jezek (2004), Madani et al. (2004),river or roads Boge (2003) or terrestrial landmarks Limet al. (2004) and it matches the detected features withcorresponding reference features taken from a database,see Fig. 1. Significant contributors to the GCP based imagenavigation errors are landmark variations over time, imagedistortions (blur, lightning), imprecise knowledge of thelandmarks absolute position and cloud coverage. Groundcontrol points are locally distributed on the Earth surfaceand can provide only a limited number of information.

? Developed within a cooperation project between EADS AstriumGmbH Germany and Technische Universitat Dresden.

Fig. 1. GCP based Image Correction

Fig. 2. Optical Flow based Image Correction

To overcome these limitations the current paper pro-poses an alternative attitude estimation approach usingthe whole image information of an imaging payload. Thewell known egomotion determination via optical flow (alsoimage flow) processing Janschek et al. (2006), Tchernykhet al. (2006) is adapted to scan images Zaunick et al.(2008). This method offers several benefits against land-mark tracking: it allows a robust and high accuracy deter-mination of the satellite attitude as well as a geometricalcorrection of the payload scan images and it offers severalvariants with different level of a priori information, seeFig. 2.

Cloud coverage and cloud motion certainly disturbs thevisual navigation estimation process. Clouds hide land-marks and they disturb the matching of image segments.Cloud motion can be interpreted as a false motion ofimage content and so mistaken as false satellite motion. InSaunders and Kriebel (1988) a series of threshold tests forAVHRR data (five spectral channels) have been developed.This algorithm has become a standard scheme for clouddetection. The use of both visible and IR channels allows acloud detection during day and night time. Based on thisstandard scheme an improved cloud detection algorithmhas been developed, which incorporates solely unregisteredand non-calibrated image data and a multispectral analy-sis for discriminating cloudy and clear sky areas.

2. SYSTEM OVERVIEW

The system developed in our work, employs an attitude es-timation system based on GEO satellite scanning imagery.The goal of the system is to estimate the 3D orientation ofthe camera coordinate frame, with respect to a global ref-erence frame or relative to elapsed attitude histories. Thesystem incorporates two types of navigational concepts:(a) the direct optical flow navigation (DOF), which usestwo consecutively acquired images for navigation purposeswithout any complementary information and (b) the vir-tual optical flow navigation (VOF), which needs some apriori information. Both proposed image navigation meth-ods use two images to compute the camera orientation.The first input image is the currently acquired cameraimage, whereas the origin of the second image differs forthe two methods. In both cases the attitude determinationis based on the image flow determination between the

first and the second image. To compute the image flowa multipoint 2D correlation has been used.

Fig. 3. System Architecture

Fig. 3 illustrates the system architecture. The imagesused in the image navigation (either DOF or VOF) areacquired by the payload (scanning imager) of the satellite.The current image and parameters (complexity dependson the method) are used to estimate the state vector ofthe camera. The state based on the image navigation isfiltered and fused with the estimated state obtained fromthe on-board navigation (AOCS). Primary (e.g. attitudeand angular velocity) and secondary states (e.g. rotationsbetween coordinate systems and camera parameters) ofthe satellite are obtained after filtering. A side effectis the improvement of the overall state estimate of thesatellite. Thus, the onboard navigation can be supportedand the complexity of the AOCS can be reduced. Basedon the improved satellite state the acquired image is geo-referenced and geometrically corrected.

3. IMAGE NAVIGATION

The proposed image navigation methods are based onevaluating the movement of the image content (opticalflow) by using a FFT block matching algorithm. Forone pair of images 1 the optical flow is determined bysubdividing both images into small fragments and 2Dcorrelation of corresponding fragments. As a result of eachcorrelation the local shift vector at the specific locationis being determined, a whole set of local shift vectorsforms an optical flow matrix. The reliability of each vectoris determined by evaluating the height of the largestcorrelation peak, the relation of the highest to the secondhighest peak and by a boundary for the maximum shiftmagnitude. The image navigation method works with twodifferent kinds of reference scans: (a) a previous scannedimage or (b) a virtual image. Both concepts are describedin the following subsections.

3.1 Direct Optical Flow Navigation

This variant does not need any a priori information andit employs the so called direct optical flow navigationbetween the current image and a previous scanned image.It gives a conclusion about the camera motion which tookplace between the acquisition instants.

The camera acquires a sequence of images during opera-tional mode. Then the optical flow is computed between1 One pair of images consists of a reference image and a currentlyacquired image.

the current acquired image and a previously acquired one.Each image also serves as input for the cloud detectionalgorithm, which determines the visibility of the earthsurface on the image. Thus, the computed optical flowfield and the cloud mask form the inputs to the motionestimation algorithm. At those areas, where the earth isvisible, the optical flow vectors are used to compute therotation between the two images. There is no relation to aglobal frame and the two attitudes are independent fromeach other and from other frames. Therefore the computedrotation corresponds to a relative rotation between the twoacquisition instants.

3.2 Virtual Optical Flow Navigation

This variant needs some a priori information: a preferablyhigh accuracy geo-referenced image, an a priori camerapose estimate and surface data of the Earth (ellipsoidalparameters or digital elevation model). Fig. 4 shows theblock diagram of the VOF navigation algorithm. In someway this method is similar to the DOF navigation, sinceone acquired image from the camera is used to computethe optical flow. But the second image is a syntheticimage derived from an image generation block. Basedon a reference image, the a priori camera pose estimate(taken from the last instance or the AOCS) and surfaceparameters of the earth, a so called virtual image isgenerated. This virtual image looks like if it was taken fromthe estimated camera pose. The principle of the imagegeneration works like a ray tracing method. Consideringthe camera characteristics for each pixel a virtual line ofsight (LOS) is computed. Based on the estimated poseof the camera and the geometrical shape of the earththe intersection points of each LOS with the earth canbe determined. Thus, each pixel of the image plane getsa corresponding earth coordinate with respect to theECEF. The grey value of each pixel is interpolated withdata from the reference image. Observing the earth fromgeostationary orbit, the geometrical shape of the earth canbe assumed as ellipsoidal.

The optical flow between the current acquired image andthe generated image is computed using a 2D cross corre-lation algorithm. Cloud contaminated areas are detectedusing the cloud detection algorithm. Based on the imageflow vectors of the visible areas the camera rotation isestimated. Since the computed rotation took place be-tween the virtual and the current image, the result givesa conclusion about the difference between the estimatedand the real camera orientation. Thus, the a priori cameraorientation estimate is corrected based on the computeddifference. The corrected camera pose can be used again togenerate a new virtual image and the accuracy can be in-creased iteratively. In most cases, the algorithm convergesafter two or three iterations and the accuracy reachesa constant value. The yaw angle determination is verysensitive and also influences the other angles. Thereforethe precision of the initial values directly drives the overallaccuracy. Thus, importing the yaw angle from the AOCSand leaving it unchanged tends to make the algorithmmore robust.

Although assuming the earth ellipsoidal, the computa-tional effort to generate the virtual images is rather huge in

Fig. 4. Principle of Virtual Optical Flow Navigation (VOF)

comparison to DOF navigation where no image generationis required. Therefore the VOF method can not be used asreal time application, unless preprocessed virtual imageswhich are saved in a database or special image processinghardware, e.g. GPUs, is used for efficient computation.

4. ROTATION ESTIMATION

The computed optical flow gives a conclusion about thecamera motion which took place between the acquisitioninstants of two images. Since the origin of the two imageshas no influence on the method how to compute themotion, the egomotion estimation described below is validfor DOF navigation as well as for VOF navigation. Therotation estimation is based on the well known egomotionequation

dx =1zA(x) · ds + B(x) · dΦ (1)

where dx denotes the image shift on the image plane, zis the z-component of the world point w.r.t. the cameraframe, ds is the infinitesimal translational motion and dΦdenotes the infinitesimal rotational motion of the camera.

Since translational motion of the camera with respect tothe ECEF (earth centered and earth fixed) frame is smalland the distance to the scene points is very large, the effectof translational image flow can be neglected for the currentapplication. Thus, (1) reduces to

dx = B(x) · dΦ (2)Equation (2) represents a linar system of equations, whichcharacterises the projection of the camera rotation ontothe image plane. If sufficient many image shift vectors dxare available, the camera rotation dΦ can be computedby solving (2). The computed rotation gives a conclusionof the satellite motion between the acquision phases of thetwo input images. For more detailles refer to Zaunick et al.(2008).

5. CLOUD DETECTION

Cloud cover and cloud motion could be misinterpretedas camera motion by the algorithm and should thereforebe detected before motion estimation. The cloud detec-tion algorithm must deal with unregistered imagery. Thedeveloped cloud detection algorithm is based on the as-sumption, that cloudy areas are colder and brighter than

0 50 100 150 200 250

0

50

100

150

200

250

300

350

Land PeakSea Peak

Fig. 5. Histogram of a cloud free coastal region (VIS8)

clear-sky areas. Thus, the use of multiple channels allowsto discriminate between clouds and clear sky areas duringday and night time. For detailed reading refer to Klink(2008).

5.1 Multispectral Cloud Detection

Multispectral cloud detection is based on rather simplethreshold tests, i.e. if the value of a pixel in one spectralchannel is greater than a predefined threshold, then thepixel is regarded as cloud contaminated.

In general two subcategories of threshold tests exist: singlechannel tests and inter-channel comparison tests for whichthe channel differences are formed:

• IR10.8 . . . cold cloud test• VIS8 . . . bright cloud test (during day only)• VIS8 . . . bright cloud test (during day only)• IR10.8 - IR12.0 . . . high cloud test• IR8.7 - IR10.8 . . . cirrus cloud test• IR3.9 - IR12.0 . . . thin cirrus test (during night only)• IR10.8 - IR3.9 . . . low cloud and fog test

Furthermore it is very probable that the close-by pixelsof a cloud infected pixel are cloudy as well. For theseneighbouring pixels the same threshold test is exclusivelyrun again but with a slightly lower/higher threshold.

5.2 Dynamic Threshold Determination

A major issue for these tests is to find appropriate thresh-old values. A common solution to this problem is to deter-mine the thresholds dynamically from the scene content.Still, dynamic threshold determination remains difficultfor coastal regions since there are usually large variationsin albedo and surface temperature.

The developed algorithm is based on the assumptionthat cloud free images (both VIS and IR) have verycharacteristic spectral histograms. An image of a cloudfree coastal region has two significant sharp peaks: asea peak (dark, cold water) and a land peak (warmer,brighter). Fig. 5 and Fig. 6 show a cloud free and cloudinfected landmark and their corresponding histograms.The algorithm will be further explained by means of aVIS8 image but it works in the same way for IR imagery.In case of IR imagery the histograms will look very similarwhile the order of peaks is inverted. Clouds are cold andtherefore dark in contrary to their bright appearance inVIS imagery.

0 50 100 150 200 250

0

50

100

150

200

250

Land Peak

Sea Peak

Cloud Trail

Dynamic Threshold

Fig. 6. Histogram of a cloud infected coastal region (VIS8)

0 20 40 60 80 100 1200

0.5

1

0 20 40 60 80 100 120-1

0

1

0 20 40 60 80 100 120-1

-0.5

0

0.5

Smoothed scaled histogram

First derivative of smoothed histogram

Second derivative of smoothed histogram

Norm

aliz

edpix

elco

unt

Norm

aliz

edpix

elco

unt

Norm

aliz

edpix

elco

unt

Pixel value

Pixel value

Pixel value

Dynamic Theshold

Fig. 7. Histogram Processing

An adequate threshold must have a higher value than theland peak and is therefore located at the foot of the landpeak and at the beginning of the cloud trail. In a first step,the sea and land peaks need to be identified. The originalhistogram is smoothed to find unambiguous peaks. It ismost likely that a histogram features three peaks: land,sea and clouds. Only scenes which are completely cloudfree and have an utter homogeneous land surface or scenesin which one of the major elements (land or sea) is totallycovered, feature two peaks. Scenes which are completelycloud covered show a single peak only.

The threshold is located at the base of the second peakwhich forms the land peak in the large majority of cases.The threshold is determined by finding the first maximumof the second derivative (greatest change in slope) ofthe smoothed histogram, see Fig 7. If the histogramfeatures one distinctive peak only the threshold is foundby multiplying the peak gray level with a static factor.

5.3 Generation of the Final Cloud Mask

Each test described in the previous sections creates a bi-nary cloud mask. Since some tests tend to mark cloud freepixels as cloud contaminated or vice versa, a confidencefactor for each test is introduced. The cloud mask ofreliable tests such as the Cold Cloud test is weighted higherthan the cloud mask created by the Spatial Coherencetest. The weighted masks are merged into a single finalcloud mask by calculating their sum. Finally a pixel is

Fig. 8. Result of Cloud Detection during daytime

Fig. 9. Result of Cloud Detection during night

regarded as cloud contaminated if its value exceeds a staticthreshold.

6. SIMULATION RESULTS

6.1 Cloud Detection

The verification of the cloud detection scheme was per-formed manually since no reference cloud masks wereavailable. The cloud conditions present in the landmarkchips and the result of the cloud detection were inspectedvisually and categorized.

Fig. 8 and 9 illustrate two examples of the multispectralanalysis during daytime and during night. The underlayedimage shows the scanned area, the detected cloud mask iscoloured in magenta. These tests have been accomplishedwith various image chips. Overall 222 chips have beeninspected. The percentaged proportion of the weatherconditions of the inspected images is depicted in Fig. 10(left chart). The results of all verification tests (during dayand night) are shown in Fig. 10 (right chart).

It can be summarized that the developed cloud detectionscheme shows good results in more than 60% percent ofthe cases independent of the daytime or weather condi-tions. The algorithm tends to mark unnecessary pixelsat daytime while only detecting a part of the clouds atnight. This is probably due to the Bright Cloud Testwhich significantly contributes to the final cloud mask.The Bright Cloud Test is sensitive in scenes acquired attwilight or dawn. In general the cloud detection scheme

2%

79%

12%

7%

No clouds

Advantageous weather conditions

Disadvantegeous cloud cover

Undefinable

5%

17%8%

7%

Useless

Clouds partly detected

Appropriate cloud detection

Unnecessary pixels detected

Undefinable

64%

Fig. 10. Result of Cloud Detection during day and night

shows weaknesses in scenes which feature regions with avery high albedo, e.g. desert (very bright) and in scenesin which either land or sea is mostly covered by clouds.In the latter case clouds are misinterpreted as land areadue to the dynamic threshold determination and thereforeclouds will not be detected completely. It works best withscenes which feature a histogram with three peaks ortwo peaks with a distinctive cloud trail. However, evenif too few or too many pixels have been marked as cloudcontaminated, the remaining visible shoreline should serveas appropriate enough input data for the following edgedetection and matching process. In just very few cases, 5%,the cloud detection algorithm returned completely falseclassification results.

6.2 Rotation Estimation

The effect of the Cloud Detection on the Image Navigationhas been tested by applying the DOF Navigation. For thissimulation real satellite imagery (MSG 2 ) has been used.The repeat cycle of the acquisition phases between twoimages is 15min. Thus, cloud motion and cloud coverageare real and not generated. An angular offset of 5 ·10−3radis added to all axes to simulate the motion of the satellite.

Fig. 11 through Fig. 13 demonstrate the influence ofcloud motion on the rotation estimation of each axis.The left charts show the image based attitude estimationwithout cloud detection. Cloud contaminated image chipsdo not move as they would move due to satellite motion.This leads to erroneous attitude estimation results. Anunambiguous relationship exists between the optical flowfield and the satellite motion after eliminating of cloudcontaminated shift vectors (right charts Fig. 11 throughFig. 13). The accuracy of the estimated angles can beimproved significantly. If no valid shift vector remains, theresult is set to zero.

7. SUMMARY

The use of payload image information helps considerablyimproving the pointing knowledge of the instrument sig-nificantly. The more information is available the betteris the performance of the determination of the line ofsight. Therefore we developed a navigation system, which2 Meteosat 2nd Generation.

20 22 24 26 28 30−1

−0.5

0

0.5

1

1.5

2

Time [sec]

roll

angl

e [m

Rad

]

without Cloud Detection

true angleestimated angle

20 22 24 26 28 30−1

−0.5

0

0.5

1

1.5

2

Time [sec]

roll

angl

e [m

Rad

]

with Cloud Detection

true angleestimated angle

Fig. 11. Effect of Cloud Detection on Image Navigationresults (roll angle), left: estimation without clouddetection, right: estimation with cloud detection

20 22 24 26 28 30−1

−0.5

0

0.5

1

1.5

2

Time [sec]

pitc

h an

gle

[mR

ad]

without Cloud Detection

true angleestimated angle

20 22 24 26 28 30−1

−0.5

0

0.5

1

1.5

2

Time [sec]

pitc

h an

gle

[mR

ad]

with Cloud Detection

true angleestimated angle

Fig. 12. Effect of Cloud Detection on Image Navigationresults (pitch angle), left: estimation without clouddetection, right: estimation with cloud detection

20 22 24 26 28 30−2

−1

0

1

2

3

4

Time [sec]

yaw

ang

le [m

Rad

]

without Cloud Detection

true angleestimated angle

20 22 24 26 28 30−2

−1

0

1

2

3

4

Time [sec]

yaw

ang

le [m

Rad

]

with Cloud Detection

true angleestimated angle

Fig. 13. Effect of Cloud Detection on Image Navigationresults (yaw angle), left: estimation without clouddetection, right: estimation with cloud detection

uses the whole image information. The experiments haveshown, that cloud coverage and cloud motion disturbs theprecise reconstruction of the camera orientation. Reliablecloud detection during day and night time is essentialto assure permanent high operation performance. Thecloud detection scheme developed in our system is basedon multispectral analysis of VIS and IR channels. Cloudcontaminated image chips are eliminated. The accuracyand reliability of the system is increased significantly.

REFERENCES

Boge, T. (2003). Bordautonome Navigation von LEO-Satelliten mittels Magnetometer- und Landmarkenmes-sungen. Ph.D. thesis, Technische Universitat Dresden,Fakultat Elektrotechnik und Informationstechnik.

Janschek, K., Tchernykh, V., and Beck, M. (2006). Perfor-mance analysis for visual planetary landing navigationusing optical flow and dem matching. In In Proceedingsof the AIAA Guidance, Navigation and Control Confer-ence, AIAA-2006-6706. Keystone, CO.

Klink, V. (2008). Implementierung eines Landmark-Tracking Verfahrens fur geostationare Beobach-tungssatelliten. Technische Universitat Dresden.

Lim, Y.J., Kim, M.G., Kim, T., and Cho, S.I. (2004).Automatic precision correction of satellite images usingthe gcp chips of lower resolution. volume 2, 1394–1397vol.2.

Liu, H. and Jezek, K.C. (2004). Automated extractionof coastline from satellite imagery by integrating cannyedge detection and locally adaptive thresholding meth-ods. International Journal of Remote Sensing, 25(5),937–958.

Madani, H., Carr, J.L., and Schoeser, C. (2004). Imageregistration using autolandmark. Geoscience and Re-mote Sensing Symposium, 2004. IGARSS ’04. Proceed-ings. 2004 IEEE International, 6, 3778–3781 vol.6. doi:10.1109/IGARSS.2004.1369945.

Saunders, R.W. and Kriebel, K.T. (1988). An improvedmethod for detecting clear sky and cloudy radiancesfrom avhrr data. International Journal of RemoteSensing, 123 – 150.

Tchernykh, V., Beck, M., and Janschek, K. (2006). Opticalflow navigation for an outdoor uav using a wide anglemono camera and dem matching. In In Proceedingsof the 4th IFAC Symposium on Mechatronic Systems,volume 4. MECHATRONICS, Heidelberg, Germany.

Zaunick, E., Janschek, K., and Levenhagen, J. (2008).Optical flow based geo satellite attitude estimation frompayload image data. In ESA Conference on Guidance,Navigation and Control Systems. ESA, Trallee, Ireland.