The use of multiple cameras applications is exploding. We're not just talking about 2 cameras for 3D or depth sensing, but 3-12 cameras for applications like drones, robotics, automotive, etc. The increasing use of multiple cameras combined with the growing use of mobile components such as apps processors, image sensors, displays, etc., being used by the broad market requires logic to connect these devices. In this presentation, Ted Marena of Microsemi explains how FPGAs can be used to leverage mobile components to aggregate a large number of MIPI CSI-2 camera interfaces.

1. Multiple MIPI
CSI-2℠ Cameras
Leveraging FPGAs
Ted Marena
Director of SoC FPGA
Marketing, Microsemi

2. Agenda
• History & adoption of MIPI CSI-2 image sensors
• MIPI CSI-2 interface primer
• FPGA usage models
• Applications for multiple MIPI CSI-2 image sensors with
FPGAs
• Summary
2

3. Evolution of Image Sensor Interfaces
Parallel CMOS interface subLVDS / HiSPi interface / etc.
In 90s & 2000s, parallel interface was the norm
In the next decade, various proprietary interfaces were introduced
~ 2-3MP

4. As mobile platforms exploded, so did CSI-2
4

5. Because of mobile popularity, everyone
drifted to CSI-2
• Just like a decade ago when PC components were used
broadly, as mobile adoption exploded, so did the
acceptance of MIPI CSI-2
5

6. Title & Date
“LP Mode” – 1.2V CMOS
“HS Mode” – SLVS200; Vcm = 200mV & Vdiff = +/- 100mV
MIPI CSI-2 D-PHY Overview

7. • Waveforms
Low
Power
Mode:
TransiUon from
Low Power to
High Speed
High Speed Mode
TransiUon from
High Speed to
Low Power
LP11, LP01, LP00, HS0, HS-SYNC …. HS0, LP-11
References the P and N channel value in LP mode respecUvely
References the value of the SLVS200 signal state
…00011101 = ‘hB8
MIPI CSI-2 Transition

8. MIPI Byte Packet Signaling
8’hB8
8’hB8
4-lane example
byte_data[7:0] {VC,DT} Byte(0) Byte(4) CS ECC Byte(n) …
… byte_clk
sync
byte_data[15:8] WC Byte(1) Byte(5) {VC,DT} Byte(0) Byte(n-1)
…
8’hB8 WC Byte(2) Byte(6) WC Byte(1) CS …
8’hB8 ECC Byte(3) CS WC Byte(2) CS …
byte_data[23:16]
byte_data[31:24]
8’b00011101 HS-SYNC Sequence
Packet Header
VC – Virtual Channel number
DT – Data Type
WC = 16 bit word length count
ECC – Checksum
on word count
CS –Checksum on
data

9. MIPI D-PHY, CSI-2 Overview
• Universal D-PHY
Title & Date
1.2V CMOS Transminer
1.2V CMOS Receiver
1.2V CMOS
ContenUon DetecUon
Circuit
SLVS200 Transminer
SLVS200 Receiver
Dynamic TerminaUon

10. D-PHY vs. C-PHY
• The majority of MIPI CSI-2 cameras use the D-PHY.
C-PHY, 3-wire, embedded clock, 3-phase coding
D-PHY, diff-pairs, source synchronous, no coding
TransiUon at every symbol, used for clock recovery

11. Emulating D-PHY in FPGA
• Normally FPGAs do not have a D-PHY hard I/O block
• They usually interface via translation resistors
11
FPGA I/Os
Image
Sensor I/Os

12. Key Blocks Used in FPGAs for Imaging
12
AP/uP/uC
Math Blocks/DSP Memory Blocks
I/O Gearing Processor/Micro

13. FPGAs in Imaging/Video Applications
Bridging is the simplest design.
Acceleration requires more performance & capabilities.
13
FPGA
A B FPGA
HW ACC
A A’
DRAM
Op9onal
Bridging
AcceleraUon

14. FPGAs Also Do Processing & Aggregating
Processing could be in an embedded processor or with FPGA fabric,
memory and math blocks (DSP blocks).
Aggregation leverages the large I/O capability of FPGAs and the fabric.
14
FPGA
Aggrega9on
A
A’
DRAM
Processing AggregaUon
FPGA
Processing
A A’
B’
DRAM
B
C

15. Why Use an FPGA?
• Not enough MIPI CSI-2 or other inputs on an AP/ISP
• Limited numbers of ISP engines on an AP/ISP
• FPGA may perform some processing allowing for a lower
cost AP/ISP
• Newer capabilities that are not available with an AP/ISP
• An FPGA can implement a complete ISP
15

16. Multi MIPI CSI-2 Camera Applications
• 3D camera / virtual reality
• Dual surveillance
• Multiple-image sensor HDR
• 180-degree surveillance
• 360-degree panorama
• Surround view automotive
• Depth-detection applications
• Drone usage
16

17. 3D Camera Example
Dual Image Sensors
Bridge
CSI-2
CSI-2
1-4
Data
Lanes
Dual CSI-2 Bridge
CSI-2
1-4
Data
Lanes
Sync and
format logic
Although ISP devices oren have mulUple camera inputs, they oren benefit from
an FPGA. Synchronizing the image sensors & arranging them

18. 3D Camera FPGA Implementation
Dual Image Sensors
Bridge
CSI-2
CSI-2
1-4
Data
Lanes
Dual CSI-2 Bridge
CSI-2
1-4
Data
Lanes
Sync and
format logic
The FPGA can arrange the image in a side-by-side or a top-bonom configuraUon.
This makes it easier for the ISP or AP to process the image.

19. Dual Surveillance Camera
Dual Image Sensors
Bridge
CSI-2
CSI-2
1-4
Data
Lanes
CSI-2
1-4
Data
Lanes
CSI-2 Tx
Although ISP devices oren have mulUple camera inputs, they oren benefit from
an FPGA that can arrange the image.
ISP
Image Buffer
Memory
Display
spliter

20. Dual Surveillance FPGA Function
This soluUon allows an ISP to process both images.
• Both images are recovered in the FPGA
• The FPGA combines the two images into
one for the ISP – oren a top-bonom
configuraUon
• This allows the ISP to process the two
images as one, but the output can be
split into two images

21. Image Sensor HDR Processing
CSI-2
May go to ISP/AP or storage
Each image sensor captures frames at exactly the
same time. A short, medium and long exposure is
used for each.
Image Buffer
Memory
FPGA with
HDR engine

22. Image Sensor HDR Processing
CSI-2
Short-, medium- and long-exposure images. Processed using local and global
tone mapping, motion artifact correction, etc.
Image Buffer
Memory
FPGA with
HDR engine

23. 180-Degree Surveillance Camera
Bridge
CSI-2
CSI-2
1-4
Data
Lanes
CSI-2 Stitching
CSI-2
1-4
Data
Lanes
CSI-2
The image sUtching funcUon is more easily done in an FPGA.
In this design, the enUre ISP could also be in the FPGA.
ISP
Image Buffer
Memory
.
.
.
3 or more

24. 180-Degree Surveillance FPGA Function
• MulUple images are recovered in the
FPGA
• The frames are stored likely in external
memory
• The FPGA performs an analysis to
determine where to merge the image
• The images are sUtched together
• Also likely a smoothing technique is used
• The image output is then processed in
the FPGA or formaned and passed onto
the ISP or AP

25. 360-Degree Cameras
CSI-2
Each image sensor frames are captured and combined.
Image processing could be in the FPGA or AP/ISP.
Image Buffer
Memory
FPGA with
sUtching,
fisheye
correcUon, etc
.
.
.
ISP or AP

26. 360-Degree Cameras
• The FPGA performs an analysis to determine
where to merge the images
• The images are stitched together
• Depending on the output format, fisheye
correction may be implemented
• The image output is then processed in the FPGA
or formatted and passed onto the ISP or AP
26

27. Surround View Automotive
CSI-2
Frame
Buffer
memory
FPGA
FPD3 Link to
MIPI CSI-2
FPD3 Link to
MIPI CSI-2

28. Surround View Application
• The FPGA implements the stitching of the images
• It formats the image for the ISP/AP
• Or FPGA processes the image and drives the display
• FPGA could add overlay such as directional lines

29. Multi Camera for Depth Detection
CSI-2
The FPGA processes all the images and
provides the lowest possible latency for
highest accuracy & response.
Image Buffer
Memory
FPGA syncs
dual camera
pairs & does
the processing
.
.
.
CSI-2
Could be one or more pairs

30. Multi Cameras for Depth-Based Analytics
Dual Image Sensors
The FPGA or AP computes 3D point cloud from a top-down stereo pair. Depth-based
analyUcs can disUnguish adults, kids, people from shopping carts for accurate people
counUng.
Top-Down
Depth
Image
FlowMetrics™ by PercepTonic

31. Depth Detection for Robotics
• The FPGA synchronizes each camera pair and processes what each
camera pair sees
• The robot can preciously determine the distance for each axis that
has a camera pair
• Parallel processing of the FPGA gives the robot the quickest
response & accuracy

32. Multi Camera Drone Application
CSI-2
These dual cameras allow a drone to “see.”
Image Buffer
Memory
FPGA syncs
dual camera
pairs, muxes,
encrypUon,
etc. .
.
.
CSI-2
GigE/10GE
Encrypted
video (wifi or
other RF)
PCIe

33. Multi Camera Drone Application
• The FPGA can merge, mux, pre-process, and run
analytics pre-processing before the AP gets involved for
3D point cloud.
Adds situational awareness so that drones do not crash into anything.

34. Summary
• Multiple camera applications continue to grow.
• MIPI CSI-2 adoption will grow and be further adopted
outside of mobile applications.
• An increasing number of FPGAs will support MIPI CSI-2
image sensors.
• The parallel architecture of the FPGA is ideal for multiple
camera applications.
34