E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Binocular StereoBinocular Stereo
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Stereo Vision [1]Stereo Vision [1]
scene pointscene point
optical centeroptical center
image planeimage plane
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Stereo Vision [1]Stereo Vision [1]
• Basic Principle: Triangulation– Gives reconstruction as intersection of two rays– Requires
• calibration• point correspondence
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Point Correspondence [1]Point Correspondence [1]
p p’ ?
Given p in left image, where can the corresponding point p’ in right image be?
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
The Simplest Case: The Simplest Case: RectiRecti--linear Configuration [1]linear Configuration [1]
• Image planes of cameras are parallel.• Focal points are at same height.• Focal lengths same.• Then, epipolar lines are horizontal scan lines.
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
The Simplest Case: The Simplest Case: RectiRecti--linear Configuration [3]linear Configuration [3]
Imageplane
),,( ZYXP
),,( ZYXP
f
Oy
x
z
ZZ
YY
XX
ZZYYXX
OPPO
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
The Simplest Case: The Simplest Case: RectiRecti--linear Configuration [3]linear Configuration [3]
Imageplane
),,( ZYXP
),,( ZYXP
f
Oy
x
z
)1,,()1,,(),,(ZYf
ZXfyxZYX
YyXxZYfY
ZXfXfZ
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
The Simplest Case: The Simplest Case: RectiRecti--linear Configuration [3]linear Configuration [3]
),,(1 ZYXP 1Oy
x
z
f
2Oy
x
z
B
BfxxZ ,,, offunction a as for expression Derive 21
1p
2p
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
The Simplest Case: The Simplest Case: RectiRecti--linear Configuration [3]linear Configuration [3]
),,(1 ZYXP 1Oy
x
z
f
2Oy
x
z
B
211
11
1
12
1
11 ,
xxBfZ
ZBfx
ZBXfx
ZXfx
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
The Simplest Case: The Simplest Case: RectiRecti--linear Configuration [1]linear Configuration [1]
OOll OOrr
PP
ppll pprr
TT
ZZ
xxll xxrr
ff
T T is the stereo baselineis the stereo baselined d measures the difference in retinal position between correspondinmeasures the difference in retinal position between corresponding pointsg points
Then given Z, we can compute X and Y.
(xl,yl)=(f X/Z, f Y/Z)(xr,yr)=(f (X-T)/Z, f Y/Z)
Disparity:d=xl-xr=f X/Z – f (X-T)/Z
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Stereo Constraints [1]Stereo Constraints [1]
X1
Y1
Z1O1
Image plane
Focal plane
M
p p’Y2
X2
Z2O2
Epipolar Line
Epipole
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Epipolar Geometry and Fundamental Matrix [1]Epipolar Geometry and Fundamental Matrix [1]
• The geometry of two different images of the same scene is called the epipolar geometry.
• The geometric information that relates two different viewpoints of the same scene is entirely contained in a mathematical construct known as fundamental matrix.
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
3D from two views [1]3D from two views [1]
Principle of binocular vision is triangulationtriangulation. Given a single image, the 3D location of any visible object point must lie on the straight line that passes through COP and image point. Intersection of two such lines from two views is triangulation.
There are two ways of extracting 3D from a pair of images. • Classical method, called calibrated route, we need to calibrate both cameras (or viewpoints) w.r.t some world coordinate system. i.e, calculate the so-called epipolar geometry by extracting the essential matrix of the system.
• Second method, called uncalibrated route, a quantity known as fundamental matrix is calculated from image correspondences, and this is then used to determine the 3D.
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Point Mapping between Images [1]Point Mapping between Images [1]
courtesy of F. Dellaert
x1 x’1
x2x’2
x3 x’3
Is there a transformation relating the points xi to x’i ?
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Point Mapping between Images [1]Point Mapping between Images [1]
• What is the relationship between the images x, x’ of the scene point X in two views?
• Intuitively, it depends on:
– The rigid transformation between cameras (derivable from the camera matrices P, P’)
– The scene structure (i.e., the depth of X)
• Parallax: Closer points appear to move more
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Baseline and Epipolar Plane [1]Baseline and Epipolar Plane [1]
• Baseline: Line joining camera centers C, C’• Epipolar plane π
: Defined by baseline and scene point X
from Hartley& Zisserman
baseline
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Epipoles and Epipolar Lines [1]Epipoles and Epipolar Lines [1]
• Epipolar lines l, l’: Intersection of epipolar plane π
with image planes
• Epipoles e, e’: Where baseline intersects image planes– Equivalently, the image in one view of the other camera center.
C C’from Hartley& Zisserman
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Epipolar Pencil [1]Epipolar Pencil [1]
• As position of X varies, epipolar planes “rotate” about the baseline (like a book with pages)– This set of planes is called the epipolar pencil
• Epipolar lines “radiate” from epipole—this is the pencil of epipolar lines
from Hartley& Zisserman
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Epipolar Constraints [1]Epipolar Constraints [1]
• Camera center C and image point x define ray in 3-D space that projects to epipolar line l’ in other view (since it’s on the epipolar plane)
• 3-D point X is on this ray, so image of X in other view x’ must be on l’
• In other words, the epipolar geometry defines a mapping x→l’, of points in one image to lines in the other
from Hartley& Zisserman
C C’
x’
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Epipolar Lines of Converging CamerasEpipolar Lines of Converging Cameras
from Hartley & Zisserman
Which is left image? How can we determine it?
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Epipolar Lines of Converging CamerasEpipolar Lines of Converging Cameras
from Hartley & ZissermanLeft view Right view
Intersection of epipolar lines = EpipoleIndicates direction of other camera
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
From Geometry to Algebra [1]From Geometry to Algebra [1]
O O’
P
pp’
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
From Geometry to Algebra [1]From Geometry to Algebra [1]
O O’
P
pp’
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
From Geometry to Algebra [1]From Geometry to Algebra [1]
Linear Constraint: Should be able to express as matrix multiplication.
Rotation arrow should be at the other end, to get p’ in p coordinates
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Matrix Form of CrossMatrix Form of Cross--product [1]product [1]
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Matrix Form of CrossMatrix Form of Cross--product [1]product [1]
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
From Geometry to Algebra [1]From Geometry to Algebra [1]
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
The Essential Matrix [1]The Essential Matrix [1]
If un-calibrated, p gets multiplied by Intrisic matrix, K
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
The Fundamental Matrix [1]The Fundamental Matrix [1]
• Mapping of point in one image to epipolar line in other image x
l’ is expressed algebraically by the fundamental matrix F
• Write this as l’ = F x• Since x’ is on l’, by the point-on-line definition we know that
x’T l’ = 0• Substitute l’ = F x, we can thus relate corresponding points in
the camera pair (P, P’) to each other with the following:x’T F x = 0
linepoint
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
The Fundamental Matrix [1]The Fundamental Matrix [1]
• F is 3 x 3, rank 2 (not invertible, in contrast to homographies)– 7 DOF (homogeneity and rank constraint take away 2 DOF)
• The fundamental matrix of (P’, P) is the transpose FT
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
In computer vision, the fundamental matrix F is a 3×3 matrix which relates corresponding points in stereo images. In epipolar geometry, with homogeneous image coordinates, x and x′, of corresponding points in a stereo image pair, Fx describes a line (an epipolar line) on which the corresponding point x′
on the other image must lie. That means, for all pairs of corresponding points holds
Being of rank two and determined only up to scale, the fundamental matrix can be estimated given at least seven point correspondences. Its seven parameters represent the only geometric information about cameras that can be obtained through point correspondences alone.
The above relation which defines the fundamental matrix was published in 1992 by both Faugeras and Hartley.
The Fundamental Matrix [2]The Fundamental Matrix [2]
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Longuet-Higgins' essential matrix satisfies a similar relationship, the essential matrix is a metric object pertaining to calibrated cameras,while the fundamental matrix describes the correspondence in more general and fundamental terms of projective geometry. This is captured mathematically by the relationship between a fundamental matrix F and its corresponding essential matrix E, which is
K and K’ being the intrinsic calibration matrices of the two images involved
The Fundamental Matrix [2]The Fundamental Matrix [2]
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Computing Fundamental Matrix [1]Computing Fundamental Matrix [1]
Fundamental Matrix is singular with rank 2
In principal F has 7 parameters up to scale and can be estimated from 7 point correspondences
Direct Simpler Method requires 8 correspondences
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Pseudo InversePseudo Inverse--Based [1]Based [1]
0uFuT
Each point correspondence can be expressed as a linear equation
01
1
333231
232221
131211
vu
FFFFFFFFF
vu
01
33
32
31
23
22
21
13
12
11
FFFFFFFFF
vuvvvvuuvuuu
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Pseudo InversePseudo Inverse--Based [1]Based [1]
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
• Input: n point correspondences ( n >= 8)– Construct homogeneous system Ax= 0 from
• x = (f11 ,f12 , ,f13 , f21 ,f22 ,f23 f31 ,f32 , f33 ) : entries in F• Each correspondence gives one equation• A is a nx9 matrix (in homogenous format)
– Obtain estimate F^ by SVD of A• x (up to a scale) is column of V corresponding to the
least singular value– Enforce singularity constraint: since Rank (F) = 2
• Compute SVD of F^• Set the smallest singular value to 0: D -> D’• Correct estimate of F :
• Output: the estimate of the fundamental matrix, F’• Similarly we can compute E given intrinsic parameters
The EightThe Eight--Point Algorithm [1]Point Algorithm [1]
0lT
r pFp
TUDVA
TUDVF ˆ
TVUDF' '
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Locating the Epipoles from F [1]Locating the Epipoles from F [1]
• Input: Fundamental Matrix F– Find the SVD of F– The epipole el is the column of V corresponding to the null
singular value (as shown above)– The epipole er is the column of U corresponding to the null
singular value• Output: Epipole el and er
TUDVF
el lies on all the epipolar lines of the left image
0lT
r pFp
0lT
r eFp
F is not identically zero
True For every pr
0leF
pl pr
P
Ol Orel er
Pl Pr
Epipolar Plane
Epipolar Lines
Epipoles
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Corner Detection [5]Corner Detection [5]Im1.jpg Im2.jpg
thresh = 500; % Harris corner threshold
% Find Harris corners in image1 and image2 [cim1, r1, c1] = harris(im1, 1, thresh, 3); [cim2, r2, c2] = harris(im2, 1, thresh, 3);
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
CorrelationCorrelation--Based Matching [5]Based Matching [5]
dmax = 50; % Maximum search distance for matchingw = 11; % Window size for correlation matching
% Use normalised correlation matching[m1,m2] = matchbycorrelation(im1, [r1';c1'], im2, [r2';c2'], w, dmax);
% Display putative matchesshow(im1,3), set(3,'name','Putative matches')
for n = 1:length(m1);line([m1(2,n) m2(2,n)], [m1(1,n) m2(1,n)])
end
Putative matches
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
RANSACRANSAC--Based Fundamental Matrix Estimation [5]Based Fundamental Matrix Estimation [5]
% Assemble homogeneous feature coordinates for fitting of the% fundamental matrix, note that [x,y] corresponds to [col, row]x1 = [m1(2,:); m1(1,:); ones(1,length(m1))];x2 = [m2(2,:); m2(1,:); ones(1,length(m1))];
t = .002; % Distance threshold for deciding outliers
[F, inliers] = ransacfitfundmatrix(x1, x2, t, 1);
Inlying
matches
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
EpipolarEpipolar
Lines [5]Lines [5]
% Step through each matched pair of points and display the% corresponding epipolar lines on the two images.
l2 = F*x1; % Epipolar lines in image2l1 = F'*x2; % Epipolar lines in image1
% Solve for epipoles[U,D,V] = svd(F);e1 = hnormalise(V(:,3));e2 = hnormalise(U(:,3));
for n = inliersfigure(1), clf, imshow(im1), hold on, plot(x1(1,n),x1(2,n),'r+');hline(l1(:,n)); plot(e1(1), e1(2), 'g*');
figure(2), clf, imshow(im2), hold on, plot(x2(1,n),x2(2,n),'r+');hline(l2(:,n)); plot(e2(1), e2(2), 'g*');
end
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Estimation of Camera Matrix [6]Estimation of Camera Matrix [6]
• Once the essential matrix is known, the camera matrices may be retrieved from E. In contrast with the fundamental matrix case, where there is a projective ambiguity, the camera matrices may be retrieved from the essential matrix up to scale and a four-fold ambiguity.
• A 3x3 matrix is an essential matrix if and only if two of its singular values are equal, and the third is zero.
• We may assume that the first camera matrix is P=[I|0]. In order to compute the second camera matrix, P’, it is necessary to factor E into the product SR of a skew- symmetric matrix and a rotation matrix.
• Suppose that the SVD of E is Udiag(1,1,0)VT. Using the notation of W and Z, there are (ignoring signs) two possible factorizations E=SR as follows:
S=UZUT, R=UWVT or UWTVT
0 1 01 0 00 0 1
W0 1 01 0 0
0 0 0
Z
ZW= 1 0 00 1 00 0 0
E=SR=(UZUT)(UWVT)= UZ(UTU)WVT
= UZWVT
= U diag(1,1,0) VT
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Estimation of Camera Matrix [7]Estimation of Camera Matrix [7]
S=UZUT, R=UWVT
or UWTVT
• For a given essential matrix E=Udiag(1,1,0)VT, and first camera matrix P=[I|0], there are four possible choices for the second camera matrix P’, namely
P’=[UWVT|u3 ] or [UWVT|-u3
] or [UWTVT|u3
] or [UWTVT|-u3
]
Where, u3
is the last column of U.
The four possible solutions for calibrated reconstruction from E.
Epipole
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Linear triangulation methods [8]Linear triangulation methods [8]
•
In each image we have a measurement x=PX, x’=P’X, and these equations can
be combined into a form AX=0, which is an equation linear in X.
•
The homogeneous scale factor is eliminated by a cross product to give three
equations for
each image point, of which two are linearly independent.
• For the first image, x×(PX)=0 and writing this out gives
Where, piT
are the rows of P.•
1
2
3
0 11 0
0
T
T
T
yx
y x
p Xp X 0p X
3 1
3 2
2 1
( ) ( ) 0( ) ( ) 0( ) ( ) 0
T T
T T
T T
xyx y
p X p Xp X p Xp X p X
3 1
3 2
3 1
3 2
' ' '' ' '
T T
T T
T T
T T
xy
xy
p pp p
Ap pp p
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Linear triangulation methods [8]Linear triangulation methods [8]
• An equation of the form AX=0
can then be composed, with
Where two equations have been included from each image, giving a
total of four
equations in four homogeneous unknowns. This is a redundant set of equations, since the solution is determined only up to scale.
X
can be calculated by SVD of A.
3 1
3 2
3 1
3 2
' ' '' ' '
T T
T T
T T
T T
xy
xy
p pp p
Ap pp p
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Fundamental matrix estimation methodsFundamental matrix estimation methods
Y
X
Z
'X
'Y
'Z
camx X 'camx
cam x PX ' 'cam x P X
0cam x PX ' ' 0cam x P X3T 1T
3T 2T
3T 1T
3T 2T
p pp p
where A'p' p'' p' p'
xy
xy
AX = 03T 1T
3T 2T
2T 1T
p p 0p p 0p p 0
xyx y
X XX XX X
3T 1T
3T 2T
2T 1T
' p' p' 0' p' p' 0' p' ' p' 0
xyx y
X XX XX X
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Stereo Rectification [1]Stereo Rectification [1]
• Image Reprojection– reproject image planes onto common
plane parallel to line between optical centers
•• Notice, only focal point of camera really Notice, only focal point of camera really mattersmatters
How can we make images as in recti- linear configuration?
Stereo Rectification
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Stereo Rectification [1]Stereo Rectification [1]
• Rectification – Given a stereo pair, the intrinsic and extrinsic parameters, find the
image transformation to achieve a stereo system of horizontal epipolar lines
– A simple algorithm: Assuming calibrated stereo cameras
p’ lp’r
P
Ol OrX’r
Pl Pr
Z’l
Y’l Y’r
TX’l
Z’r
Stereo System with Parallel Optical Axes
Epipoles are at infinity
Horizontal epipolar lines
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
Stereo Rectification [1]Stereo Rectification [1]
• Algorithm– Rotate both left and
right camera so that they share the same X axis : Or -Ol = T
– Define a rotation matrix Rrect for the left camera
– Rotation Matrix for the right camera is RrectRT
– Rotation can be implemented by image transformation
pl pr
P
Ol Or
Xl
Xr
Pl Pr
Zl
Yl
Zr
Yr
R, T
TX’l
Xl ’ = T_axis, Yl ’ = Xl ’xZl , Z’l = Xl ’xYl ’
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
The Fundamental Matrix Song [4]The Fundamental Matrix Song [4]
The fundamental matrix
Used in stereo geometry
A matrix with nine entries
It's square with size 3 by 3
Has seven degrees of freedom
It has a rank deficiency
It's only of rank two
Call the matrix F and you'll see...
Two points that correspond
Column vectors called x and x-prime
x-prime transpose times F times x
Equals zero
every time
The epipolar constraint
Involves epipolar lines
Postmultiplying
F by x
Results in vector l-prime
It's the epipolar line
In the other view passing through x-prime
A three component vector
Of homogeneous design
The left and right nullspaces
of F
Are the epipoles
e-prime and e
All of the epipolar lines
Should pass through these
Here's a linear estimation example:
Take a set of 8 point samples
Construct a matrix, take the SVD
And the elements of F are in the last column of V
If you try to estimate
F with a coplanar set of points
Your sample set will be degenerate
And will not bring you joy
When doing the estimation
If you don't perform rank deprivation
Your epipolar lines
And the epipoles will not coincide
But if your scene has three views
The trifocal tensor
is what you'd use
Constraints from the third view act like glue
That can't be determined from just two views
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
ReferencesReferences
1. Chandra Kambhamettu, “Multipleview1,” Delaware University Lecture Material of Computer Vision (CISC 4/689), 2007.
2. Wikipedia, “Fundamental Matrix (Computer Vision),” http://en.wikipedia.org/wiki/Fundamental_matrix_(computer_vision)
3. Sebastian Thrun, Rick Szeliski, Hendrik Dahlkamp and Dan Morris, “Stereo1,” Stanford University Lecture Material of Computer Vision (CS223B), Winter 2005.
4. Daniel Wedge, “The Fundamental Matrix Song,” http://danielwedge.com/fmatrix/5. Peter Kovesi, “Example of finding the fundamental matrix using RANSAC,” available
at http://www.csse.uwa.edu.au/~pk/Research/MatlabFns/Robust/example/index.html
6. Richard Hartley and Andrew Zisserman, “8.6 The essential matrix,” Multiple View Geometry in Computer Vision, Cambridge, pp. 238-241.
7. Richard Hartley and Andrew Zisserman, “11.2 Linear triangulation methods,” Multiple View Geometry in Computer Vision, Cambridge, pp. 238-241.
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
3D Simulation3D Simulation
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
1. Motion Field: Camera Model1. Motion Field: Camera Model
• Making a projection matrix P using related parameters.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%f=12*0.001; % 12mmImagerSize=1/3; % 1/3 inchHres=640; Vres=480;u0=320; v0=240;
%%%%%%%%%%%%%%%%%%%%%%%%%%%%PixelSize=(ImagerSize*0.0254)*(Hres/sqrt(Hres^2+Vres^2))/Hres;ku=1/PixelSize; kv=1/PixelSize;
au=-f*ku; av=-f*kv;
%%%%%%%%%%%%%%%%%%%%%%%%%%%%Mint=[au 0 u0 0; 0 av
v0 0; 0 0 1 0];Mext=[[RotationMatrix(0,0,0) [0 0 0]']; [0 0 0 1]];P=Mint*Mext;
• 1inch=25.4mm• Imager size is the
diagonal length of the imager
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
1. Motion Field: Initial Image1. Motion Field: Initial Image
• Making a test pattern and its image.
N=21;X=zeros(N,N,3);Z0=3;d=0.1;for i=1:N
for j=1:Nx=d*(i-round(N/2));y=d*(j-round(N/2));X(j,i,:)=[x,y,Z0];
endend
Xh=[reshape(X,[N*N,3]) ones(N*N,1)]';xh0=P*Xh;x0=zeros(2,size(xh0,2));x0(1,:)=xh0(1,:)./xh0(3,:);x0(2,:)=xh0(2,:)./xh0(3,:);
figure; axis([1 640 1 480]); hold on;plot(x0(1,:),x0(2,:),'b+');
100 200 300 400 500 600
50
100
150
200
250
300
350
400
450
-1-0.5
00.5
1
-1-0.5
0
0.512
2.5
3
3.5
4
plot3(X(:,:,1),X(:,:,2),X(:,:,3),'b.');
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
2. 3D Reconstruction: Two Projection Matrix2. 3D Reconstruction: Two Projection Matrix
• Defining each projection matrixe of two cameras.
Mint=[au 0 u0 0; 0 av
v0 0; 0 0 1 0];
Mext1=[[eye(3) [0 0 0]']; [0 0 0 1]];P1=Mint*Mext1;
R2=RotationMatrix(0,-0.2,0.3);T2=[0.3 0 -1]';Mext2=[[R2 T2]; [0 0 0 1]];P2=Mint*Mext2;
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
-2
0
2
-3 -2 -1 0 1 2 3
0
1
2
3
4
5
6
7
8
9
10
yx
z
2. 3D Reconstruction: 3D Objects2. 3D Reconstruction: 3D Objects
• Making two 3D objects, cubes.
Cube=[1 -1 -1
1 1 -1 -1
1; 1 1 -1 -1
1 1 -1 -1; -1 -1
-1
-1
1 1 1 1];R1=RotationMatrix(-10/180*pi, 20/180*pi, 45/180*pi);Cube1=0.3*R1*Cube+[0.5*ones(1,8); 0.8*ones(1,8); 5*ones(1,8)];R2=RotationMatrix(20/180*pi, -30/180*pi, -15/180*pi);Cube2=0.5*R2*Cube+[-0.1*ones(1,8); -0.5*ones(1,8); 7*ones(1,8)];
figure; hold on;DrawCube3D(Cube1);DrawCube3D(Cube2);axis equal;axis([-3 3 -3 3 0 10]);xlabel('x'); ylabel('y'); zlabel('z'); grid on;
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
x
y
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
2. 3D Reconstruction: DrawCube2D2. 3D Reconstruction: DrawCube2D
function DrawCube2D(cube)line([cube(1,1) cube(1,2)],[cube(2,1) cube(2,2)]);line([cube(1,2) cube(1,3)],[cube(2,2) cube(2,3)]);line([cube(1,3) cube(1,4)],[cube(2,3) cube(2,4)]);line([cube(1,4) cube(1,1)],[cube(2,4) cube(2,1)]);
line([cube(1,1) cube(1,5)],[cube(2,1) cube(2,5)]);line([cube(1,2) cube(1,6)],[cube(2,2) cube(2,6)]);line([cube(1,3) cube(1,7)],[cube(2,3) cube(2,7)]);line([cube(1,4) cube(1,8)],[cube(2,4) cube(2,8)]);
line([cube(1,5) cube(1,6)],[cube(2,5) cube(2,6)]);line([cube(1,6) cube(1,7)],[cube(2,6) cube(2,7)]);line([cube(1,7) cube(1,8)],[cube(2,7) cube(2,8)]);line([cube(1,8) cube(1,5)],[cube(2,8) cube(2,5)]);
plot(cube(1,1),cube(2,1),'r.');plot(cube(1,2),cube(2,2),'g.');plot(cube(1,3:8),cube(2,3:8),'b.');
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
2. 3D Reconstruction: DrawCube3D2. 3D Reconstruction: DrawCube3D
function DrawCube3D(Cube)line([Cube(1,1) Cube(1,2)],[Cube(2,1) Cube(2,2)], [Cube(3,1) Cube(3,2)]);line([Cube(1,2) Cube(1,3)],[Cube(2,2) Cube(2,3)], [Cube(3,2) Cube(3,3)]);line([Cube(1,3) Cube(1,4)],[Cube(2,3) Cube(2,4)], [Cube(3,3) Cube(3,4)]);line([Cube(1,4) Cube(1,1)],[Cube(2,4) Cube(2,1)], [Cube(3,4) Cube(3,1)]);
line([Cube(1,1) Cube(1,5)],[Cube(2,1) Cube(2,5)], [Cube(3,1) Cube(3,5)]);line([Cube(1,2) Cube(1,6)],[Cube(2,2) Cube(2,6)], [Cube(3,2) Cube(3,6)]);line([Cube(1,3) Cube(1,7)],[Cube(2,3) Cube(2,7)], [Cube(3,3) Cube(3,7)]);line([Cube(1,4) Cube(1,8)],[Cube(2,4) Cube(2,8)], [Cube(3,4) Cube(3,8)]);
line([Cube(1,5) Cube(1,6)],[Cube(2,5) Cube(2,6)], [Cube(3,5) Cube(3,6)]);line([Cube(1,6) Cube(1,7)],[Cube(2,6) Cube(2,7)], [Cube(3,6) Cube(3,7)]);line([Cube(1,7) Cube(1,8)],[Cube(2,7) Cube(2,8)], [Cube(3,7) Cube(3,8)]);line([Cube(1,8) Cube(1,5)],[Cube(2,8) Cube(2,5)], [Cube(3,8) Cube(3,5)]);
plot3(Cube(1,1),Cube(2,1),Cube(3,1),'r.');plot3(Cube(1,2),Cube(2,2),Cube(3,2),'g.');plot3(Cube(1,3:8),Cube(2,3:8),Cube(3,3:8),'b.');
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
2. 3D Reconstruction: Image onto Camera12. 3D Reconstruction: Image onto Camera1
• Making image onto camera1 using perspective transformation.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Camera1에
대한
image 만들기%%%%%%%%%%%%%%%%%%%%%%%%%%%%Xh1=[Cube1; ones(1, 8)];th1=P1*Xh1;t1=zeros(2,size(th1,2));t1(1,:)=th1(1,:)./th1(3,:);t1(2,:)=th1(2,:)./th1(3,:);
Xh2=[Cube2; ones(1, 8)];th2=P1*Xh2;t2=zeros(2,size(th2,2));t2(1,:)=th2(1,:)./th2(3,:);t2(2,:)=th2(2,:)./th2(3,:);
x1=[t1 t2];
figure; axis([1 640 1 480]); hold on;DrawCube2D(t1);DrawCube2D(t2);
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
2. 3D Reconstruction: Image onto Camera22. 3D Reconstruction: Image onto Camera2
• Making image onto camera2 using perspective transformation.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Camera2에
대한
image 만들기%%%%%%%%%%%%%%%%%%%%%%%%%%%%th1=P2*Xh1;t1=zeros(2,size(th1,2));t1(1,:)=th1(1,:)./th1(3,:);t1(2,:)=th1(2,:)./th1(3,:);
th2=P2*Xh2;t2=zeros(2,size(th2,2));t2(1,:)=th2(1,:)./th2(3,:);t2(2,:)=th2(2,:)./th2(3,:);
x2=[t1 t2];
figure; axis([1 640 1 480]); hold on;DrawCube2D(t1);DrawCube2D(t2);
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
2. 3D Reconstruction: Generated Stereo Images2. 3D Reconstruction: Generated Stereo Images
100 200 300 400 500 600
50
100
150
200
250
300
350
400
450
100 200 300 400 500 600
50
100
150
200
250
300
350
400
450
Camera1 Camera2
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
2. 3D Reconstruction: Camera Parameter Estimation2. 3D Reconstruction: Camera Parameter Estimation
%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Fundamental matrix와
camera matrix를
추정한다.%%%%%%%%%%%%%%%%%%%%%%%%%%%%tol
= .001; % Distance threshold for deciding outliers
[F, inliers] = ransacfitfundmatrix(x1, x2, tol, 1);
K=Mint(:,1:3);E=K'*F*K;[U,S,V]=svd(E);W=[0 -1 0; 1 0 0; 0 0 1];Z=[0 1 0; -1 0 0; 0 0 0];
S=U*Z*U';R_=U*W'*V';
P1_=K*[eye(3) [0 0 0]']P2_=K*[R_ U(:,3)];
K
is required if 3D reconstruction will be done with image coordinates.
Selecting the best from 4 candidates by hand.
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
2. 3D Reconstruction: Triangulation2. 3D Reconstruction: Triangulation
• In each image we have a measurement x=PX, x’=P’X, and these equations can
be combined into a form AX=0, which is an equation linear in X.• An equation of the form AX=0 can then be composed, with
Where two equations have been included from each image, giving a
total of four
equations in four homogeneous unknowns. This is a redundant set of equations, since the solution is determined only up to scale.
• X
can be calculated by SVD of A.
3 1
3 2
3 1
3 2
' ' '' ' '
T T
T T
T T
T T
xy
xy
p pp p
Ap pp p
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
2. 3D Reconstruction: Triangulation2. 3D Reconstruction: Triangulation
%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 3차원
정보
복원하기%%%%%%%%%%%%%%%%%%%%%%%%%%%%N=size(x1,2);
X_=zeros(3,N);
for n=1:NA=[x1(1,n)*P1_(3,:)-P1_(1,:);...
x1(2,n)*P1_(3,:)-P1_(2,:);...x2(1,n)*P2_(3,:)-P2_(1,:);...x2(2,n)*P2_(3,:)-P2_(2,:)];
[U,S,V]=svd(A);Xh_=V(:,4);X_(:,n)=Xh_(1:3)./Xh_(4);
end
Cube1_=X_(:,1:8);Cube2_=X_(:,9:16);
figure; hold on;DrawCube3D(Cube1_);DrawCube3D(Cube2_);axis equal;axis([-3 3 -3 3 0 10]);xlabel('x'); ylabel('y'); zlabel('z'); grid on;
• Notice that the 3D reconstruction is done 3D point by point.
3 1
3 2
3 1
3 2
' ' '' ' '
T T
T T
T T
T T
xy
xy
p pp p
Ap pp p
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
2. 3D Reconstruction: Reconstruction Result2. 3D Reconstruction: Reconstruction Result
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
x
y
-3 -2 -1 0 1 2 3-3
-2
-1
0
1
2
3
xy
Original Objects Reconstructed Objects
E-mail: [email protected]://web.yonsei.ac.kr/hgjung
2. 3D Reconstruction: Reconstruction Result2. 3D Reconstruction: Reconstruction Result
• Notice that the result is up to scale.
-2
0
2
-3 -2 -1 0 1 2 3
0
1
2
3
4
5
6
7
8
9
10
yx
z
Original Objects Reconstructed Objects
-2
0
2
-3 -2 -1 0 1 2 3
0
1
2
3
4
5
6
7
8
9
10
yx
z