MIPI DevCon 2016: Image Sensor and Display Connectivity Disruption
MIPI DevCon 2016: Accelerating Software Development for MIPI CSI-2 Cameras
-
Upload
mipi-alliance -
Category
Mobile
-
view
295 -
download
9
Transcript of MIPI DevCon 2016: Accelerating Software Development for MIPI CSI-2 Cameras
Accelerating Software Development for MIPI
CSI-2℠ Cameras
Licinio Sousa Synopsys, Inc.
The Challenge Developing SW for Multiple HW Configs
• Nowadays HW platforms are always evolving • It is hard for SW to keep up with the changes • Using a monolithic approach might be simple to begin,
but harder to mantain • To support a new device, the designer then needs to either split the
code or add support for another device in the initial code • After a while the code will become “spaghetti code” • Each device has specific configuration requirements and
characteristics, which complicates the tasks of providing support and integration, due to limited code reusability
• Steep learning curve
2
The Challenge Developing SW for Multiple HW Configs
3
CompleteMIPICameraandDisplaySolu6on
The Solution Leverage Linux Resources to Reduce Design Efforts
• Systems with embedded Linux can leverage existing libraries, APIs and driver frameworks to reduce effort and accelerate SW development
• V4L2 API and modular design approach • Media controller • Device tree
Agenda
• Introduce a few Linux concepts and resources • Share our experience • Conclusion
Linux
# facts - Linux is a free and open source software ~18 000 000 lines of code - 5.64 patches accepted in mainline kernel / h - More than 1000 engineers work in each release
# who is working in Linux - Only 17.9 % make kernel hacking for free - The other 82.1% is made by top tech players
Linux
WearablesSmartphones
Servers
Mainframes
Embeddedsystems
When is Linux’s Birthday?
8
From:[email protected](LinusBenedictTorvalds)Newsgroup:comp.os.minixSubject:GCC-1.40andaposixques6onMessage-ID:1991Jul13,[email protected]:3Jul9110:00:50GMTHellonetlanders,DueaprojectI'mworkingon(inminix),I'minterestedintheposixstandarddefini6on.Couldsomebodypleasepointmetoa(preferably)machine-readableformatofthelatestposixrules?Ftp-siteswouldbenice.LinusTorvaldstorvalds@kruuna.helsinki.fi
From:[email protected](LinusBenedictTorvalds)
Newsgroup:comp.os.minix
Subject:Whatwouldyouliketoseemostinminix?
Summary:smallpollformynewoperaIngsystem
Message-ID:1991Aug25,[email protected]
Date:25Aug9120:57:08GMT
OrganizaIon:UniversityofHelsinki.
Helloeverybodyoutthereusingminix-I'mdoinga(free)operaIngsystem(justa
hobby,won'tbebigandprofessionallikegnu)for386(486)ATclones.
(…)AnysuggesIonsarewelcome,butIwon'tpromiseI'llimplementthem:-)
From:[email protected]
elsinki.FI(LinusB
enedictTorvalds)
Newsgroups:com
p.os.minix
Subject:Freeminix-likek
ernelsourcesfor3
86-AT
Message-ID:<1991
Oct5.054106.4647
@klaava.Helsinki.F
I>
Date:5Oct9105:
41:06GMT
Organiza6on:Uni
versityofHelsink
i
Doyoupineforth
enicedaysofminix-1.1,
whenmenweremenandw
rotetheirown
devicedrivers?Ar
eyouwithoutan
iceprojectandju
stdying
tocutyourteeth
onaOSyoucantr
ytomodifyforyournee
ds?Areyoufindin
git
frustra6ngwhene
verythingworkso
nminix?Nomoreall-
nighterstogeta
ni`yprogramworking
?Thenthispostm
ightbejustforyou
:-)
AsImenIonedamonth(?)a
go,I'mworkingonafre
eversionofaminix-look
alike
forAT-386computers.It
hasfinallyreach
edthestagewhe
reit'sevenusabl
e
(thoughmaynotbe
dependingonwh
atyouwant),
andIamwillingt
oputoutthesou
rcesforwiderdistr
ibuIon.Itisjust
version0.02
(+1(verysmall)patch
already),butI've
successfullyrunb
ash/gcc/gnu-make/gnu-
sed/compressetc
underit. Thereareactuallythree
"birthdays"onecanidenIfyforLinux!!
Inside Linux – Architecture
User Space vs. Kernel Space
• User Space • If SW is running in userspace it’s usually called an Application • Memory area where application software will run
• Kernel Space • Strictly reserved for running privileged OS kernel • Runs kernel extensions and device drivers • A driver has direct access to the HW
User-Kernel Interface
• User-kernel interaction is made via Linux APIs • Linux provides APIs in
order to allow programmers to write applications consistent with the operating environment
• Some APIs can be divided in 2 parts, which run in two different areas • Kernel space • User space
Video4Linux2 API API for Video capture
• One of the Linux APIs is V4L (Video4Linux) a.k.a. V4L2 • V4L2 is an API for supporting real-time video capture in
Linux • V4L2 is used to build drivers for many USB webcams,
TV tuners, smartphone cameras, and related devices • With V4L2, applications can easily add video support in
a standard way
V4L2 API
KernelSpace
ApplicaIonSpace
V4L2LinuxDrivers
V4L2IOCTLs
mplayer ffmpeg vlc
V4L2IOCTLs
Hardware
Hardware
V4L2 Devices and Subdevices Modular Approach to Model the HW
• Hardware is often made of several integrated circuits that need to interact with each other • This can lead to complex
drivers. • The drivers must
reflect the hardware model in software • Modelling the different
hardware components as software blocks called sub-devices.
V4L2device
V4L2sub-device
V4L2sub-device
V4L2sub-device V4L2sub-device
Media Controller API to Define Relation Between Device/Sub-Devices
• Allows to discover a platform configuration and configure it
• Devices are described as building blocks • Connected through pads
• Pads can be either sinks or sources • No limit for pads and sinks on each entity
Device/Sub-device and Media Entity Source & Sink Pads to Represent Data Flow
Media Device - HW Pipeline V4L2 Device/Sub-Devices Structured Using
MediaController API Med
ia
Device
CameraCSI-2Host
Colorspaceconverter
VideoDevice
/dev/video
Device Driver Interaction Only High-Level Interaction is Needed
• The main driver will call the sub-device drivers to • power on, • start streaming, • change video mode, • and many more.
• All this without needing to know anything about the HW it is talking to
VideoDevice
/dev/video mplayer
KernelSpace
ApplicaIonSpace
Device Tree Describes the HW Configuration
• The format is expressive and able to describe most board design aspects including • the number and type of CPUs • base addresses and size of RAM • busses and bridges • peripheral device connections • interrupt controllers and IRQ line
connections • and others
Device Tree Hardware Details Listed
• The HW specific details will be available in the Device Tree
• HW addresses, IRQs and more are always available in the Device Tree
• In the case of V4L2 cameras, information like number of lanes, frequency and others is also available
V4L2, Media Controller and Device Tree Putting it All Together
21
V4L2, Media Controller and Device Tree Putting it All Together
• Device driver is designed using V4L2 API – API for video capture
• It is modular and captures the HW elements – Sub-devices
• Media controller API is used to define the relationship between the device and sub-devices, forming a HW pipeline that represents the actual HW platform
• Device tree describes the HW platform and individual block specific details (address, interrupts, version, number of lanes)
22
V4L2, Media Controller and Device Tree Putting it All Together
• When kernel starts, it will look at the device tree and configure it self • The sub-device drivers will be loaded if present in the device tree
and will get the necessary details from it • The device driver HW pipeline will be formed
• Application interacts with video device using high level commands • The video device is accessible under /dev • The application will open /dev/videoX and interact using the high
level commands (such as change resolution, shutdown, power on, start capture, etc.)
• Commands will be translated into low-level commands, specific to each individual sub-device (CSI-2 controller, camera, etc.)
23
V4L2, Media Controller and Device Tree Putting it All Together
• If a change in the platform is required (replace camera or controller, etc.) • The new individual sub-devices need to be added to the kernel • The device tree needs to be updated and recompiled to reflect the
HW changes • By re-starting the Kernel, the Device driver HW pipeline will be
updated automatically
24
OUR EXPERIENCE
CSI-2 SW Development The Media Device HW Pipeline
26
CSI-2 SW Development The Device Tree
Camera
PHY
VideoDMA
CSIController
CSI-2 SW Development Simple to Change CSI-2 Camera
Camera
MIPI CSI-2 Host IP Prototyping Kit
• Synopsys offers an integrated HW/SW solution that takes advantage of all these concepts
VisitSynopsysboothtoseedemo!
Leverage Kernel Tools to Accelerate CSI-2 SW Development
• Application interacts with driver only using high level Commands • No need to use specific commands for each HW element
• The same kernel can serve for several HW platforms by simple changing the Device Tree • No effort in recompiling the kernel
• Modular code is easily split among team members • Each driver just needs to provide standard interaction commands
• Easy to maintain and add support for new HW
All this accelerates driver development by reducing the code complexity and turnaround time!
30
Thank You!
Synopsys’broadporkolioofDesignWareMIPIIPsoluIonssynopsys.com/mipi