Large-Scale Point Cloud Classification Benchmark · 2016. 8. 15. · IGP & CVG, ETH Zürich Timo Hackel, [email protected] | 7/6/2016 | 25 Voxelization details 16x16x16

||IGP & CVG, ETH Zürich

www.semantic3d.net

7/6/2016www.semantic3d.net, [email protected] 1

Large-Scale Point Cloud Classification

Benchmark


People


Timo

Hackel

Nikolay

Savinov

Ľubor

Ladický

Jan Dirk

Wegner

Konrad

Schindler

Marc

Pollefeys


Terrestrial Laser Scanning

Architecture, Archaeology , City Modelling, Mining,

Monitoring, …


~200.000.000 $ annually spend for

terrestrial laser scanners

Measure up to 2.000.000 3D points

per second

Laser scanner


Objective

Assign a class label to each 3D point individually



Related Benchmark Tests

[1] Vallet, Bruno, et al. "TerraMobilita/iQmulus urban point cloud analysis benchmark."

Computers & Graphics 49 (2015): 126-133.

[2] Silberman, Nathan, et al. "Indoor segmentation and support inference from RGBD

images." Computer Vision–ECCV 2012. Springer Berlin Heidelberg, 2012. 746-760.


iQmulus & TerraMobilita NYU

||IGP & CVG, ETH Zürich 7/6/2016www.semantic3d.net, [email protected] 6


Data

{x, y, z, intensity, r, g, b}



Intensity

External illumination sensor used to remove external illumination (sun, etc.)

Value range: -2047 to + 2048


Color

Color was recorded after scanning


Cube maps


Scanning Artefacts


Classes

Man made terrain

Natural terrain

High vegetation

Low vegetation

Buildings

Clutter (hard scape)

Scanning artefacts

Cars


training and test set

Over 3 billion labelled points



Concept

“reduced-8”

reduced challenge

“semantic-8”

full challenge

training set:

1.660.295.483 points

15 scans

unfiltered

test set:

78.699.329 points

3 scans

voxel grid filter:

1 cm resolution

test set:

2.348.935.300 points

15 scans

unfiltered


How to submit your results: ww.semantic3d.net

1. Create zip file containing the results as described in “submit”

2. login

3. Create a new classifier

4. Upload data

5. Choose if you want to make results public

1. 2.

3.4. 5.


Class distribution


0100200300400500600700800900

Train, million

Test, million


Evaluation measures


Main: intersection over union (IoU)

Averaged over all classes

Well-established for segmentation:

e.g. PASCAL VOC

If cij = count [class i predicted as j]

then:

Auxiliary measure: accuracy

i

i

j

k


Human labeller disagreement ≤ 4%


00.0050.01

0.0150.02

0.0250.03

0.0350.04

0.045

1 - (intersection over union)


Baseline methods


Covariance

FeaturesPure Color Deep

Learning


Baseline

Pure Color


||IGP & CVG, ETH Zürich 7/6/2016Timo Hackel, [email protected] 21

Cube mapping


Result

Reduced challenge

Full challenge


Baseline II

Deep Learning on voxelized neighbourhood


Voxel occupancy

grid: any point

inside voxel? (0/1)

3D Convolutional

Neural Network

Goal: classify

center of

neighbourhood

(sliding 3D window)

Semantic label


Baseline II

Deep Learning on voxelized neighbourhood


Classification & detection, no segmentation

3D ShapeNets [CVPR’15]

VoxNet [IROS’15]

Deep Sliding Shapes [CVPR’16]

…

Prior work:


Voxelization details

16x16x16 voxels neighbourhood

5 scales considered: voxel size 2.5 cm -

0.8 m

Scales concatenated

Batch constructed on C++ side

Batch transferred to Torch via Luajit

occupancy hash table (sparse)

classified x, y, z query

neighbourhood (dense)

…


Deep net details

VGG-like

Multi-scale

Torch

Input scale 1

Input scale 5

…

VGG-like 1

VGG-like 5

… Concatenation

of FCFC

Predictions

VGG 3x3x3

kernel


Training details & results

xy-rotation augmentations necessary

z direction aligned with gravity

Batch size 100, every 100 batches random xy-rotation

Trained on 1 point cloud with 259 million points

Classes sampled randomly with equal probabilities

Test results: DeepNet better than 2D color classification

x y

z

7/6/2016Timo Hackel, [email protected] 27


Code available soon!

C++/Lua

You could build your algorithm on top of that.

Will be on benchmark website.

Subscribe to our newsletter!


Baseline III

traditional machine learning with multiscale features

7/6/2016Timo Hackel, [email protected] 29

compute

neighborhoods

extract

features

classify

semantically

X=2

Y=3.1 Y=1.5

p1 p2 p3 p4

RFX Y


Neighborhood approximation

KD-trees slow

Further approximation needed

Multiscale voxel-grid filtering

KD-tree pyramid


Feature extraction

Spherical neighborhood

Cylindrical neighborhood


Training

4. Detect and remove unused features

1. Subsampling of training set x to generate x’

2. Grid search and cross-validation depth d of random forest

3. Train random forest using d and x’

class frequencies per scan do not necessarily

represent prior distribution over class labels

Error weights depend on distance to scanner

Subsampling when scanner origin unknown


Classification & Implementation details

Precompute KD-Trees and

keep them in RAM

Evaluate features on the fly

(nearly no RAM needed)

Implementation in C++ with

OpenMP

Solve eigenvalues and

eigenvectors of 3x3 Matrix

analytically


Results on Mobile Mapping Data


Results on Mobile Mapping Data


Result

Reduced challenge

Full challenge


Why does the confusion matrix not contain all points

from the submitted result?

Why ASCII and not LAZ?

Why is the training set of semantic-8 and reduced-8

the same?

What are the next steps?


Questions and answers


Demo of baseline III

Documents

Large-Scale Point Cloud Classification Benchmark · 2016. 8. 15. · IGP & CVG, ETH Zürich Timo Hackel, [email protected] | 7/6/2016 | 25 Voxelization details 16x16x16