View
28
Download
0
Category
Preview:
Citation preview
Seeing the Invisibles:
A Backstage Tour of Information Forensics
Include joint research with Wei-Hong Chuang, Adi Hajj-Ahmad, Ravi Garg, Hongmei Gou, Shan He, K.J. Ray Liu, Christine McKay, Hui Su, Ashwin Swaminathan, Wade
Trappe, Avinash Varna, Jane Wang, Chau-Wai Wong, and Hong Zhao.
MMiinn WWuu
Media and Security Team (MAST) ECE Department / UMIACS
University of Maryland, College Park
http://www.ece.umd.edu/~minwu/research.html
A Decade+ of Research on Info. Forensics
! Our life and work are forever changed … – By advances in electronics, computing, communications, sensing,
and signal processing
! So much multimedia info and content right at our hand … – Keep us entertained
– Used as important evidences & records
! Gather traces of evidence – Origin, history, integrity, etc.
– A broad range of applications
Min Wu (UMD): Multimedia Information Forensics 2
Role Play as Sherlock Holmes in our Digital Era
1. Leak: A proprietary image sent to 10 people got leaked out
" Who leaked the info?
Min Wu (UMD): Multimedia Information Forensics 3
Role Play as Sherlock Holmes in our Digital Era
1. Leak: A proprietary image sent to 10 people got leaked out
" Who leaked the info?
2. Source: Picture of a heavily guarded xPhone7 prototype showed up on web
" Is it a real photo or a graphic rendition? Who in the company took it using his/her camera?
3. When/Where: an audio clip with incriminating words showed up
" were its content and recording time true as claimed?
Min Wu (UMD): Multimedia Information Forensics 4
Role Play as Sherlock Holmes in our Digital Era
1. Leak: A proprietary image sent to 10 people got leaked out
" Who leaked the info?
2. Source: Picture of a heavily guarded xPhone7 prototype showed up on web
" Is it a real photo or a graphic rendition? Who in the company took it using his/her camera?
3. When/Where: an audio clip with incriminating words showed up
" were its content and recording time true as claimed?
Min Wu (UMD): Multimedia Information Forensics 5
Role Play as Sherlock Holmes in our Digital Era
1. Leak: A proprietary image sent to 10 people got leaked out
" Who leaked the info?
2. Source: Picture of a heavily guarded xPhone7 prototype showed up on web
" Is it a real photo or a graphic rendition? Who in the company took it using his/her camera?
3. When/Where: an audio clip with incriminating words showed up
" were its content and recording time true as claimed?
Min Wu (UMD): Multimedia Information Forensics 8 8 8
White House
Satellite Image
Alice
Bob
Carl
w1
w2
w3
Leak
Example: Replacing the Last Bit of Pixels
Replace LSB with Pentagon s MSB
UM
CP
EN
EE
631
Slid
es (
crea
ted
by M
.Wu
© b
ased
on
Res
earc
h Ta
lks
98-
04)
Min Wu (UMD): Multimedia Information Forensics 13 Min Wu (UMD): Multimedia Information Forensics 15
Robust Watermarking via Spread Spectrum Embedding
◆ Embedding domain tailored to media characteristics & application requirement
10011010 …
© Copyright …
Min Wu (UMD): Multimedia Information Forensics 16
Fingerprinting Topographic Map
– Traditional protection: intentionally alter geospatial content – Embed much less intrusive digital fingerprint for a modern protection
• 9 long curves are marked; 1331 control points used to carry the fingerprint
1100x1100 Original Map Fingerprinted Map
Min Wu (UMD): Multimedia Information Forensics 17
Collusion-Resistant Fingerprinting: Examples
2-User Interleaving Attack 5-User Averaging Attack
. . .
=> Can survive combination attacks of collusion + print + scan and detect fingerprint from hard copies
Min Wu (UMD): Multimedia Information Forensics 18
Joint Coding-Embedding via Anti-Collusion Codes
Extracted fingerprint code ( -1, 0, 0, 0, 1, …, 0, 0, 0, 1, 1, 1 )
Collude by Averaging Uniquely Identify User 1 & 4
Embed fingerprint via HVS-based spread spectrum embedding
16-bit ACC for detecting up to 3 colluders out of 20
User-1 ( -1,-1, -1, -1, 1, 1, 1, 1, …, 1 ) ( -1, 1, 1, 1, 1, 1, …, -1, 1, 1, 1 ) User-4
19
Applications: From Government to Hollywood
Insert special signals to identify recipients – Deter leak of proprietary documents
– Consider imperceptibility, robustness, traceability
White House
Satellite Image
Alice
Bob
Carl
w1
w2
w3
Leak
Rights management by copyright industry ($500+Billion ~ 5% U.S. GDP)
! Successfully catch media pirates => Wide adoption now – Alleged Movie Pirate Arrested by FBI – Oscar screening copies (Jan. 2004) – Track down illicit sharing of digital TV access – Daqing, China (Nov. 2007):
found and convicted perpetrator
Carmine Caridi Russell friends … Internet w1 Last Samurai
Hollywood studio traced pirated version
Explore More …
! Information Forensics: An Overview of the First Decade, by M. Stamm, M. Wu, K.J.R. Liu, invited article for inaugural issue, IEEE Access, May 2013.
– Joint Coding and Embedding; fundamental tradeoffs
– Large scale video fingerprinting; special media type such as maps
– Group based fingerprinting; behavior forensics
! Recent directions: probabilistic fingerprint/tracing code
! Industry R&D: by movie industry (e.g. Technicolor R&D, MovieLab) and printing/marketing (e.g. Digimarc Inc.)
Min Wu (UMD): Multimedia Information Forensics 20
Role Play as Sherlock Holmes in our Digital Era
1. Leak: A proprietary image sent to 10 people got leaked out
" Who leaked the info?
2. Source: Picture of a heavily guarded xPhone7 prototype showed up on web
" Is it a real photo or a graphic rendition? Who in the company took it using his/her camera?
Min Wu (UMD): Multimedia Information Forensics
Metadata in header can be forged #
22
Min Wu (UMD): Multimedia Information Forensics 23
! Break down the info. processing chain into individual components
! Identify algorithms and parameters employed in major components of a digital device or processing system
! Concept extensible to general info proc. chain beyond multimedia
Exploit Intrinsic Fingerprints via Component Forensics
Color Filter Array (CFA)
Color Interpolation
White Balancing
Real world scene Digital image Camera Components
… Sensors …
Intrinsic Traces Representing Group Properties
! Digital / software components of device or processing system
! Ensemble properties of analog components: e.g. statistical noise profile of sensors
CFA Interpolation
R ?
? ?
R ?
? ?
R ?
? ?
R ?
? ?
? ?
? ? Candidate
CFA pattern
24 Digital photograph Scanner model 1 Scanner model 2
Fitting error
Min Wu (UMD): Multimedia Information Forensics 24
Min Wu (UMD): Multimedia Information Forensics 26
Photography vs Computer Graphics?
*Photorealistic image source – Columbia univ. database
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
PF
PD
PF – prob. of false alarm (% of photographic images misclassified)
PD – prob. of correct decision (% of computer generated images classified correctly)
Intrinsic Traces Link to Individual Devices
! Individual variation from analog part of sensors – Unreproducible properties due to manufacturing variability
! Challenges to overcome – Picture content variability, post-processing, anti-forensics, etc.
27 LG VX9700 #1 LG VS9700 #2 Canon SX230
43 -7 12 -36
-9 -10 65 51
71 -9 40 101
53 -13 54 74
-14 -27 -5 -42
65 37 56 63
95 68 41 -11
61 40 2 -51
-123 -106 -34 16
63 -11 -4 39
60 59 16 -15
-83 64 7 -20
Note: PRNU pattern amplified by a factor of 42,500
Matching Sensor Noise via Correlation Metrics
! Measure similarity by normalized cross-correlation (NCC) – NCC gives a sharp peak at the right alignment of right match
– May include weights from measurement reliabilities
! Robust peak measurement – Normalize with ambient
correlation values (Fridrich et al.)
match
mis-match
30 Min Wu (UMD): Multimedia Information Forensics Min Wu (UMD): Multimedia Information Forensics 31
Tampering Detection !!
Explore intrinsic fingerprints left by various processing modules – To infer the algorithms and parameters employed in various components of the
digital device and processing systems
– New traces or vanished old traces suggests potential post-camera operations
Estimated coeff.
From direct camera output
After post- camera filtering
Color Interpolation
Color Sensors
Scene Optical Lens System CAMERA
Other Software Processing
A Tampering
/ Stego
B
Black: Sony P72; White: Canon Powershot S410 Grey: Classified as other cameras with low confidence
Explore More …
! Early years’ research:
Special Issue on Digital Forensics by IEEE Signal Processing Magazine (March 2009) Edited by E. Delp, N. Memon and M. Wu
and a concurrent special issue in IEEE Security & Privacy Magazine
! Information Forensics: An Overview of the First Decade, by M. Stamm, M. Wu, K.J.R. Liu, IEEE Access (invited), May 2013.
! New DARPA “MediFor” program
Min Wu (UMD): Multimedia Information Forensics 32
Role Play as Sherlock Holmes in our Digital Era
1. Leak: A proprietary image sent to 10 people got leaked out
" Who leaked the info?
2. Source: Picture of a heavily guarded xPhone7 prototype showed up on web
" Is it a real photo or a graphic rendition? Who in the company took it using his/her camera?
Min Wu (UMD): Multimedia Information Forensics 34
Role Play as Sherlock Holmes in our Digital Era
1. Leak: A proprietary image sent to 10 people got leaked out
" Who leaked the info?
2. Source: Picture of a heavily guarded xPhone6 prototype showed up on web
" Is it a real photo or a graphic rendition? Who in the company took it using his/her camera?
3. When/Where: an audio clip with incriminating words showed up
" were its content and recording time true as claimed?
Min Wu (UMD): Multimedia Information Forensics 35 Min Wu (UMD): ENF for Media Information Forensics
Ubiquitous Forensic Fingerprints from Power Grid
$ Electric Network Frequency (ENF): 50 or 60 Hz nominal Change'slightly'due'to'demand1supply'
Main'trends'consistent'in'same'grid'
37 Source: US Grid image is from InTech online
49.5 50 50.5 51 51.5
100
200
300
400
500
Frequency (in Hz)
Tim
e (in
sec
onds
)
-100
-80
-60
-40
-20
Power ENF signal
Min Wu (UMD): ENF for Media Information Forensics
Ubiquitous Forensic Fingerprints from Power Grid
49.5 50 50.5 51 51.5
100
200
300
400
500
Frequency (in Hz)
Tim
e (in
sec
onds
)
-100
-80
-60
-40
-20
-30 -20 -10 0 10 20 30
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Time frame lag
Cor
rela
tion
coef
ficie
nt
$ Electric Network Frequency (ENF): 50 or 60 Hz nominal Change'slightly'due'to'demand1supply'
Main'trends'consistent'in'same'grid'
$ ENF can bee “seen” or “heard” in sensor recordings Power'grid'influences'electronic'sensing'(E/M'interference,'vibra@on'etc)'
Help'determine'recording'@me/loca@on,'detect'tampering,'etc.'
ENF matching result demonstrating similar variations in the ENF signal extracted from video and from power signal recorded in India
Video ENF signal Power ENF signal Normalized correlation
9.6 10 10.4 10.8
100
200
300
400
500
600
Frequency (in Hz)
Tim
e (in
sec
onds
)
-150
-140
-130
-120
-110
-100
-90
-80
38
Tampering Detection
! Adding a clip into original video leads to discontinuity in ENF – Clip insertion can also be detected by comparing the video ENF
signal with the power ENF at corresponding time
160 320 480 640 800 960
10
10.1
10.2
10.3
Time (in seconds)
Freq
uenc
y (in
Hz)
ENF signal from Video
160 320 480 640 800
49.9
50
50.1
50.2
Time (in seconds)
Freq
uenc
y (in
Hz)
Ground truth ENF signal
Inserted clip
39 Min Wu (UMD): Multimedia Information Forensics
“Forensic Binding” of Audio and Visual Tracks
$ High'correla@on'of'ENFs'in'audio'&'video'captured'at'same'@me'=>'can'extend'to'synch'mul@ple'media'streams'
'
''
'''''
$ An@1Forensic'Study:''possible'to'remove'narrow1band'ENF;'but'much'harder'to'tamper/transplant'a'valid'ENF'w/o'being'caught''
'
'' ' ' ' ' ' ' ' ' ' ' ' ' '' ' ' ' ' ' ' ' ' ''' ' '' ' ' ' ' ' '
40 Min Wu (UMD): ENF for Media Information Forensics
From Time Stamps to Location
! Match with ENF references over times + grids – Verify or exhaustively search for
the matching location on grid level
41
What if no concurrent references available? – Explore overall characteristics
of ENF in a grid
– Also reduce computation of exhaustive search
Source: US Grid image is from InTech online
Freq
uenc
y E
stim
ates
(H
z)
Freq
uenc
y E
stim
ates
(H
z)
Time (secs) Time (secs) Time (secs)
Freq
uenc
y E
stim
ates
(H
z)
Explore Machine Learning to Infer Location
! Inter-Grid location-of-recording estimation from sensing signals containing ENF traces
– Identified useful features for average 94% accuracy on audio
LEBANON INDIA Eastern US
Min Wu (UMD): Multimedia Information Forensics 42
From Time Stamps to Location
! Match with ENF references over times + grids – Verify or exhaustively search for
the matching location on grid level
43
What if no concurrent references available? – Explore overall characteristics
of ENF in a grid
– Also reduce computation of exhaustive search
Can we determine where within a grid? – E.g. DC or NYC in US East?
– Look at subtle traces in ENF
– Relate ENF correlation with distance
Source: US Grid image is from InTech online
Can ENF Pinpoint to Locations Within a Grid?
! Main trend of ENF is known to be the same in a grid
! Microscopic traces – Aggregated effect of local events and propagations from elsewhere
! Our multi-location studies in U.S. east and west grids – Relate pairwise ENF correlations between query and anchor points
with geographic and wireline distances
Min Wu (UMD): Multimedia Information Forensics 44 (a) ENF signals from different locations of US
Eastern grid (b) Correlation between ENF signals
after high-pass filtering
ENF in Historical Recordings
! Two ENFs may appear in digitized tape recordings 1) original ENF; and
2) ENF at time of digitization
=> Provide digital preservation guidelines to better utilize invisible traces
! Distortions and artifacts – Drifting; low SNR; etc.
! Ongoing: create a historical ENF database – Timestamp recordings of
historic importance
47
Digitized Kennedy White House recording (1962 Cuban missile crisis)
NASA audio from Apollo 11
(1969 1st moon-landing)
/16 ACM – Immersive Multimedia 2014: ENF Signal for Video Synchronization
Speed Restoration: ENF as Intrinsic Freq. Reference
NASA Apollo 11 Mission recording Before Restoration After Restoration
Spectrogram
Estimated ENF Estimated ENF
/16 ACM – Immersive Multimedia 2014: ENF Signal for Video Synchronization
Immersive Media: Synch Streams via ENF in Audio
Demo-2: Videos at different locations of Lab Building
Video after synchronization 2 synched stopwatches (as ground truth)
Demo-1: Videos in Gym
(different viewing angles)
ENF Research & References
Min Wu (UMD): ENF for Media Information Forensics 59
Estimate ENF Signal (instantaneous freq.): Robust, high resolution;
Exploit harmonics
SPL’13. APSIPA ’12, ACM MM’11, TIFS’13
Visual Modality: Handle aliasing – exploit rolling shutters;
Handle motion
TIFS’13, ICIP’14, ACM MM’14 Immersive Media
Modeling & Analysis: Statistically modeling of ENF signals;
Anti-Forensics.
WIFS’12, CCS’12, twoTIFS’13
Novel Applications: Location & integrity; Stream alignment;
Digital humanity (historic audio) ICASSP’12-13, WIFS’13 / TIFS’15, iConf’14, ACM MM’14 Immersive
Information Forensics: An Overview of the First Decade, by M. Stamm, M. Wu, K.J.R. Liu, IEEE Access, invited article for inaugural issue, May 2013.
MediFor: Newly Launched DARPA Program on media forensics aiming at restoring “Seeing is Believing”
See also Special Issue on Digital Forensics by IEEE Signal Processing Magazine (March 2009) Edited by E. Delp, N. Memon and M. Wu; and a concurrent special issue in IEEE Security & Privacy Magazine
Min Wu (UMD): Multimedia Information Forensics 61
63
THANKS to many who paved ways …
Min Wu (UMD): Multimedia Information Forensics 64
Include joint work with collaborators: K.J. Ray Liu, Wade Trappe, Jane Wang, Hong Zhao; Kari Klaus, Douglas Oard.
Min Wu (UMD): Multimedia Information Forensics
65
Include joint work with graduate & undergrad students Wei-Hong Chuang, Ravi Garg, Hongmei Gou, Adi Hajj-Ahmad, Shan He,
Christine McKay, Hui Su, Ashwin Swaminathan, Avinash Varna, Chau-Wai Wong.
Min Wu: Info Forensics & Media Security • Provide assurance for proper use of content
– Answer who has done what, when and how.
• Cross-disciplinary and balancing theory & practice – Analytic modeling for fundamental understanding – Design effective and efficient algorithms with synergy
from signal proc., comm, machine learning, crypto …
Tracing leaked documents
Device & environment fingerprints Learn more from this
URL
Recommended