User-centric Protection and Privacy in Smart

Surveillance Systems

Hauke Vagts1,2, Erik Krempel2, and Jürgen Beyerer1,2

1 Vision and Fusion Laboratory, Karlsruhe Institute of Technology, Karlsruhe,Germany.

2 Fraunhofer Institute of Optronics, System Technologies and Image ExploitationIOSB, Karlsruhe, Germany

{hauke.vagts|erik.krempel|jürgen.beyerer@iosb.fraunhofer.de}

Abstract. During the last decades surveillance systems developed fromanalog one camera one monitor systems to highly complex distributedsystems with heterogeneous sensors that can handle surveillance tasksautonomously. With raising power and complexity, ensuring privacy be-came a key challenge. An event-driven SOA architecture that follows theprivacy by design principle is a promising approach to realize a smartsurveillance system and is described in this work.

Enforcement of privacy is not only complex for engineers and system de-signers; rather it is not understandable for the average user, who cannoteven assess potentials and limitations of smart surveillance systems. Thiswork presents an approach for privacy that is focused on the user, i.e.,the observed subject. By using a mobile device the user can interact withthe surveillance system and is not passive anymore, as in conventionalsurveillance deployments. This restores the balance between the observedand the observers, enhances transparency and will raise the acceptanceof surveillance technology.

In the highlighted approach the user can control his individual-relateddata and privacy preferences and can use services that are bene�cial forhim.

1 Introduction

The overall number of surveillance systems is still increasing. In the UK, morethan 4,000,000 cameras have been deployed. Modern surveillance systems areenhanced with acoustic sensors or can read RFID tags. Simultaneously, new al-gorithms for identi�cation, risk detection and automated data processing appear.Observed people got scared by powerful and complex systems. In [1] Hempel etal. point out that 40% of the European Society thinks that CCTV invades pri-vacy. Even if acceptance highly di�ers between countries, surveillance must bemore transparent to achieve trust of the users and to restore the balance betweendata controllers and data subjects.

Smart surveillance services can also provide a bene�t for the observed objects,e.g., people might want to book a surveillance service when walking home in the

2

dark or special observation for their car in public parking. Also non-securityrelated services as inhouse navigation are possible. They all require active com-munication between the user and the surveillance system.

This work is structured as follows. Section two gives an overview about pri-vacy protection technologies in conventional surveillance systems, smart surveil-lance systems and adjacent areas. Section three speci�es an architecture for asmart surveillance system that is based on the privacy by design principle. Sec-tion four integrates a new system for user interaction, evaluated in detail insection �ve. The last section gives a short overview of the achievements andshows potential �elds for further research.

2 Recent Work

With the recent increase of video surveillance systems, privacy protection insurveillance has become an active �eld of study. More information about intelli-gent video surveillance can be found in, e.g., [2][3]. Most research work consid-ers conventional video surveillance systems with CCTV cameras and improvesprivacy by disguising regions of interest (ROI). The work from Dufaux andEbrahimi [4] uses private key encryption to scramble ROIs. Privileged users,e.g., law-enforcement authorities possess the needed private key to unscramblethe video, while unauthorized users can only view the distorted version of thecontent. Another approach from Schi�er et al. [5] proposes a practical real-timesystem that blurs the faces of people wearing special markers. This enables selec-tive privacy protection, e.g., for employees, while the camera observation staysusable for its surveillance task.

More extensive privacy approaches for video completely prevent the visu-alization of video streams. In [6] Fleck and Straÿer use smart cameras, whichare capable of processing data, to detect events in an observed area, such as afalling person. Only these events, secured by encryption, are transmitted to acentral server for visualization. The operator is completely decoupled from thelive video stream and can only see the resulting movements and events within amap. Similar work was done by Senior et al. [7]. Their system processes videostreams and constructs an abstract model of the observed environment. Accessto this model is regulated by privacy rules, enabling, e.g., anonymous access tostatistical data and law-enforcements authorities to receive the original streamwith additional annotations.

All these systems try to increase privacy by blocking access to certain data,mask the identity of the user and secure the communication channels. The in-tegration of the user was forgotten and the user had only a �take it or leave it�choice when he was confronted with a surveillance systems. This results in ac-tivist projects, like i-SEE3 of the Institute for Applied Autonomy. Their systemgenerates a route between two points in Manhattan with the least amount ofpassed surveillance systems possible.

3 http://www.appliedautonomy.com/isee.html; 2012-04-12

3

New approaches have to focus on the surveillance subject and its fears anddemands on a surveillance system.

To gather new approaches for transparency and acceptance, research in theareas of pervasive and ubiquitous computing must be considered. These systemshave many similarities with modern surveillance systems, i.e., a big amount ofpersonal data is collected and di�erent services access the data to perform certaintasks. As these systems were designed to serve individual users and not to secureareas, privacy concerns are more in focus of development.

In [8] Langheinrich proposes a system, called PAWS, to process data witha high level of transparency for the users. Whenever a user enters a new area,a privacy beacon informs him about services in place and what personal datathey process. The user can set his own trust in the services and decides, whichservices are allowed to process his personal data, so that he can use their func-tionality. The shortcoming of this system is that it completely relies on socialand legal norms to prevent the misuse of private data, since there is no systemwide control over the distributed data. Bagués et al. [9] extend the work andsuggest a complete architecture for privacy protection in pervasive computingenvironments. They consider three stages of data lifetime: collection, access andsecondary use and use di�erent techniques to prevent misuse.

Contrary to pervasive environments, main object of surveillance system isto ensure security. Hence, transparent and accepted surveillance systems muststill process certain data without permission of the user, while still giving theusers the maximum right for privacy. This makes the design of privacy awaresurveillance systems even more complex.

3 An Architecture for Privacy in Smart Surveillance

Systems

To cope with the requirements for smart surveillance systems and to ensureprivacy of observed subjects, a new architecture for smart surveillance was de-veloped. More details can be found in [10]. Four main components realize aprivacy preserving smart surveillance system. The task-oriented approach en-sures that data is only captured and processed when needed. A surveillance taskcan be broken down in multiple steps and is executed through an event-driven

SOA architecture. The captured data, from heterogeneous sensors, is stored inan abstract representation inside the Object-Oriented World Model, where it canbe accessed for further processing. The Privacy Manager ensures, that data isaccessed in a way that agrees with the desired privacy level.

3.1 Task-Oriented Approach for Surveillance

Most existing surveillance systems operate in a sensor-oriented way, i.e., manysensors are deployed in the monitored area and all available data is collected andstored. After the transmission to a central storage, intelligent algorithms processthe data to extract as much relevant information as possible. A big number of

4

Fig. 1. An architecture for a smart surveillance system

sensors directly results in a big amount of data and ine�ective systems, as mostof the data is not required. In addition, this has a huge impact on privacy. Incontrary, task-oriented systems avoid the collection of data whenever possible.Data is only gathered and processed for a speci�c purpose/task. This has to twobene�ts. Fewer resources, e.g., bandwidth are required and less sensitive datais gathered and processed. An example for a task is the observation of a singleperson from its current position to a speci�ed destination. Only one person istracked, instead of all persons in the area.

3.2 An Event-Driven SOA Architecture for Surveillance

A system following the task-oriented approach must be capable of executingsmall pieces of functionality (services) and combing them to complex work�ows.

5

A smart system also runs multiple surveillance tasks in parallel for di�erentusers. To extend the functionality of smart surveillance systems, it should beeasy to integrate new sensors and services. An event-driven SOA controls thecollaboration of the services. Services can either be signal-oriented, if they en-capsulate sensors and gain information about the environment, or can processinformation. The architecture is shown Figure 1. Services generate events, e.g.,person identi�ed, �ght detected etc., which are received by the task executionengine that coordinates all involved components. Triggered by such events otherservices are started. Depending on the scenario an operator can integrate di�er-ent kinds of prior knowledge. In the evaluation scenario, the system has multipleprior knowledge sources such as the current weather conditions to recalibrateits sensors and a geo data repository with information about the building andprohibited areas, etc.

Di�erent kinds of Signal Processing Services (SPS) are used to interpret datacaptured by sensors. Services exist to, e.g., track persons over multiple camerasor to identify a person with biometric data or other means. The Sensor Registryprovides information about existing SPS and connected sensors, e.g., �eld of viewof used cameras. The preprocessed data is delivered to the Object-Oriented WordModel (OOWM) that acts as central data storage for further processing in thesurveillance system.

Information Processing Services (IPS) process the data stored in the OOWM.Typical examples are the �Person Counter Service�, which checks that not toomany people are in a certain area or the �Area Protection Service� which sendsan alarm when people enter a restricted area.

User Level Services (ULS) provide additional services to the surveillancesubjects. They even can be integrated by third parties and are therefore notcompletely trustworthy. To cobe with law requirement, ULS are detached fromthe core processing system. Typically ULS are activated by users themself andelaborate privacy related check of the used data are performed. In Chapter 4ULS are looked at in detail.

3.3 Object-Oriented World Model

The Object-Oriented World is the central data store in the system. It trans-forms all observations from SPS into a consistent object representation and actsas high-level information source for IPS and ULS. The OOWM performs thefollowing tasks:

� Information Representation: The OOWM o�ers an application-independentrepresentation of real world objects. It distinguishes di�erent types of ob-jects, e.g., persons, luggage, animals and stores them for later access.

� Data Association: For each new object observation the OOWM decides,whether it corresponds to a new object, or if an existing object must beupdated.

� Data Fusion and Tracking: Updated information from new observations isfused with previously assessed information.

6

Fig. 2. Privacy Manager

� Data Access: Di�erent types of services need an easy way to access the storeddata. Therefore every object is represented by a unique ID and can be polledas a list of attributes with their corresponding values.

More details about the technical implementation of the OOWM can be found in[11].

3.4 Privacy Manager

The task-oriented approach reduced the capture of privacy concerning data andallows controlled processing. But it is not addressing all legal issues and notcapable to o�er the required level of transparency. The Privacy Manager (PM)is designed to enforce privacy respecting access to the OOWM and enable trans-parency for the users. ULS can only access data from inside the surveillancesystem via the PM. All access from ULS to the OOWM, or similar data repre-sentation in other smart surveillance system, is managed by the PM. It groupsdi�erent Privacy Enhancing Technologies (PET) in a single place. Thereby thesystem is very �exible and new methods can be easily integrated. The currentimplementation realizes the following components and is sketched in Figure 2:

� Identity Management (IdM): In a privacy preserving surveillance system,every component should only get access to required data. To prevent servicesto exchange data several security methods exist. One is the use of pseudo IDsfor all objects. When two or more services access data of the same object,they know the object with di�erent IDs, preventing the services from mergingattributes. More Details can be found in [12].

7

� Policy Management: Multiple modules use policies for specifying con�gu-rations, e.g., XACML access control policies, IdM policies or policies foranonymization. The Policy Management controls the access to these poli-cies for users. It also merges user speci�c policies with system policies. Moredetails can be found in [13].

� Access and Usage Control: To enforce privacy requirements Access Controlis integrated in the PM. Due to expressiveness and extendibility for privacyneeds, the PM utilizes XACML. By deploying, changing or revoking policies,the access rights of a service can be altered at runtime. More details can befound in [13]. Current research aims at integrating Usage Control, to ensurethat access to data can be controlled, even when it has left the system.

� Anonymization Modules: Besides simple permit or deny decisions, XACML iscapable of obligation handling. It is possible to write policies that only grantaccess to attributes after they have been anonimized. The PM uses di�er-ent kinds of anonymization modules. A typical job would be to anonymizeposition data to reach certain k-Anonymity or l-Diversity values [14].

� Subject Interaction: The Subject Interaction module handles the interactionbetween an observed subject, the users, and the surveillance system. Di�erentoptions for interaction with a surveillance system exist. Section 5 explainsuser interaction based on a mobile device in detail.

� Additional Privacy Modules: The PM also performs additional tasks, likekey-distribution and management, which are implemented in additional mod-ules.

4 Data Access for User Level Services

User Level Services are a new approach for surveillance systems and are not yetfully explored. Especially, the opportunity to allow third parties the developmentof their own ULS leads to a high complexity. So far three di�erent types of ULScan be distinguished:

� Security ULS accessing user data: Users can request additional security re-lated services. These services o�er a higher level of security than the coresystem, but are only activated on request of the user.

� Convenience ULS accessing user data: Each user can start ULS processinghis data for additional services, e.g., in-house navigation.

� ULS accessing anonymized data: Some ULS operate on anonymized datato provide statistical information, e.g., a service generating a heat map of ashopping area for shop operators. When processing proper anonymized data,no risk for privacy exists, but especially with multiple possible data sources,it is hard to decide whether a su�cient level of anonymization is reached ornot.

This work only addresses services accessing user data. Further research is re-quired for ULS accessing anonymized data for statistical reasons.

8

Fig. 3. Data Access for ULS

4.1 User Level Service Communication

Figure 3 shows the user-centric approach for data management. When a userwishes to use a ULS, he needs a way to interact with the surveillance system at�rst. this work proposes (smart) mobile phones. They provide enough resourcesfor complex computations and with the increasing sales, it can be assumed thatmost people will own one in future. The smart phone allows to interact with thesystem in many di�erent way, e.g., to review, which ULS are running, to managepersonal privacy policies or to start and stop ULS. When a user started an ULS,a generic communication takes place:

1. The ULS registers at the Privacy Manager (PM) and sends a data request.The PM then collects all requested data from the OOWM.

2. XACML based privacy policies are applied to the collected information. Ei-ther a default policy or personal policies managed by the concerned users.Afterwards, every attribute that the ULS is not allowed to access is re-moved[13].

3. If the privacy policies include obligations for further anonymization of thedata the Privacy Manager applies its corresponding anonymization mod-ule[14].

4. The Identity Management replaces the identi�ers from the OOWM with newunique identi�ers only used for this ULS [12].

5. In the last step, the Privacy Manager updates its intern database with servicenames and used data. This database is used to show to the users, whichservices access their data.

9

(a) Help gesture (b) Operator view of the alarm

Fig. 4. Gesture Recognition

Only after performing all steps the Privacy Manager provides the requesteddata to the ULS. The user can review the stored data corresponding to him. Hisprivacy policies also let him stay in charge of his personal data.

5 Evaluation

The proposed privacy preserving techniques and the new user-centric approachare evaluated in the Network Enabled Surveillance and Tracking (NEST) [10]demonstrator system. Multiple cameras are deployed throughout the entire siteof Fraunhofer IOSB. The operator has access to a screen that shows pixelatedvideo streams. These streams allow him to detect crime or accidents, but arenot �ne-grained enough to identify the employees. Additionally, the operatorcan access a terminal showing alarm events. In a demonstration setup, threeexemplary technologies show the potential of the new user-centric surveillanceand privacy technologies. At �rst, users can send an alarm to the operator byperforming a certain gesture. Then a security related ULS is shown to increasethe surveillance level in a certain area. The last scenario shows a conveniencerelated ULS for indoor navigation.

5.1 Gesture Recognition

The �rst example task is a gesture recognition system. This technology is part ofthe core system and data access is not regulated by the PM, due to the the highsecurity value and the little privacy impact. An information processing serviceanalyses the video streams and tries to detect a special gesture. If one is detectedthe operator receives an alarm event and access to the non-pixelated steam ofthe camera that detected the gesture. This allows users easy contact to securitywhen feeling in danger or seeing others in danger.

10

(a) Barcode detected (b) Map displaying Bob's position

Fig. 5. WatchMe Application

5.2 WatchMe Smartphone App

The second realized example is a security related ULS. The general idea of theapplication is that sometimes people want a higher level of surveillance, e.g., atan empty subway station at night. Bob might feel threatened and wants extraattention of the surveillance system and its operator. Therefore he starts the�WatchMe� application and demands a higher surveillance level by pressing abutton. After that, the system sends a special bar code to his device. Bob has topresent this barcode to the next camera (see Figure 5a) to start his surveillancetask. As soon as the system has detected the barcode the mobile device vibratesto notify the user about the successful log in. At the same time the operator seesa marker for Bob's position and his movement trace in the area (see Figure 5b).The operator can now pay extra attention to Bob.

As soon as Bob wants the extra surveillance to stop he can use his smartphone to cancel the surveillance task. The system deletes all collected informa-tion of the WatchMe task. The same happens when Bob leaves the observedarea.

5.3 Indoor Navigation

The last example shows a pure convenience ULS in action. Navigation in anunknown buildings can be hard. Therefore the system o�ers an indoor navigationsupport. At �rst, the user has to pick his desired destination. Afterwards thesystem sends a special bar code to his device, which needs to be presented tothe next camera. The app loads a privacy policy in the system that allows the

11

(a) User chooses his destination (b) Map with the shortest path

Fig. 6. Inhouse navigation

usage of the user's position for the navigation purpose. Therefore the ULS cancalculate the shortest path to the user's destination. This path gets plotted insidea map of the building and is transmitted to user.

When the user reaches his destination or wishes to cancel the tracking, thesystem deletes all collected data and the default privacy policies are reactivated.

6 Conclusion and Future Work

Complex surveillance systems can be realized by an event-driven SOA, follow-ing the task-based approach. To realize privacy by design, multiple PETs areintegrated in the Privacy Manager. Even if the PETs are very promising froma technological perspective, users do not really understand them and cannotdistinguish between them, which lead to a user-centric approach for privacyenforcement that realizes a communication channel between the user and thesystem. He is now in charge of his privacy preferences and can also use addi-tional services o�ered by the system. Transparency is enhanced and it is easy forthe users to control the system and his privacy setting. Further user studies arenecessary to see, if the user-centric approach is really enhancing the acceptanceof surveillance technology and if it is understood by the user. The results fromthe evaluation in the demonstrator system are very promising and have shownthat the new system is capable to protect user data inside a surveillance sys-tem. It must still be investigated, if it can be used in practice. Therefore imageprocessing and video analysis must improve �rst.

Still, a lot of research needs to be done in the �eld of privacy preservingsmart surveillance. More PETs must be evaluated, especially for non-video data,e.g., audio information. The current demonstration system is going to extendwith more PETs and further demonstration scenario are realized.

12

References

1. Hempel, L., Töpfer, E.: CCTV in Europe. Final Report, Urbaneye Working PaperNo. 15 (August 2004)

2. Regazzoni, C., Fabri, G., Vernazza, G.: Advanced video-based surveillance systems.Volume 488. Springer (1999)

3. Remagnino, P.: Video-based surveillance systems: computer vision and distributedprocessing. Springer Netherlands (2002)

4. Dufaux, F., Ebrahimi, T.: Scrambling for video surveillance with privacy. In:Computer Vision and Pattern Recognition Workshop (CVPRW'06), IEEE (2006)160

5. Schi�, J., Meingast, M., Mulligan, D., Sastry, S., Goldberg, K.: Respectful cameras:Detecting visual markers in real-time to address privacy concerns. In Senior, A.,ed.: Protecting Privacy in Video Surveillance. Springer (2009) 65�89

6. Fleck, S., Straÿer, W.: Smart camera based monitoring system and its applicationto assisted living. Proceedings of the IEEE 96 (2008) 1698�1714

7. Senior, A., Pankanti, S., Hampapur, A., Brown, L., Tian, Y.L., Ekin, A., Connell,J., Shu, C.F., Lu, M.: Enabling video privacy through computer vision. Security& Privacy, IEEE 3 (June 2005) 50�57

8. Langheinrich, M.: A privacy awareness system for ubiquitous computing environ-ments. In: UbiComp 2002: ubiquitous computing: 4th International Conference,Springer (2002) 315�320

9. Bagüés, S., Zeidler, A., Klein, C., Valdivielso, C., Matias, I.: Enabling personalprivacy for pervasive computing environments. Journal of Universal ComputerScience 16(3) (2010) 341�371

10. Moÿgraber, J., Reinert, F., Vagts, H.: An architecture for a task-oriented surveil-lance system � a service and event based approach. In: Fifth International Confer-ence on Systems ICONS. (April 2010)

11. Bauer, A., Emter, T., Vagts, H., Beyerer, J.: Object oriented world model forsurveillance systems. In Elsner, P., ed.: Future Security: 4th Security ResearchConference, Fraunhofer Verlag (October 2009) 339�345

12. Vagts, H., Krempel, E., Beyerer, J.: Privacy enforcement by identity managementin smart surveillance systems. In: Proceedings of the International Conferenceon Distributed Multimedia Systems. Number 16, Knowledge Systems InstituteGraduate School (October 2010) 64�69

13. Vagts, H., Krempel, E., Fischer, Y.: Access controls for privacy protection in per-vasive environments. In: The 3rd Workshop on "Privacy and Security in PervasiveEnvironments PSPAE. (2011)

14. Vagts, H., Bier, C., Beyerer, J.: Anonymization in intelligent surveillance systems.In: New Technologies, Mobility and Security (NTMS), 2011 4th IFIP InternationalConference on, IEEE (2011) 1�4

Year:2012

Author(s):Vagts, Hauke; Krempel, Erik; Beyerer, Jürgen

Title:User-centric protection and privacy in smart surveillance systems

DOI: 10.1007/978-3-642-33161-9_36 (http://dx.doi.org/10.1007/978-3-642-33161-9_36)

The original publication is available at springerlink.com

Details:Aschenbruck, N. (Ed.); Martini, P.; Meier, M.; Tölle, J.:Future Security. 7th Security Research Conference 2012. Proceedings : Bonn, Germany, September 4-6, 2012Berlin: Springer, 2012 (Communications in computer and information science 318)ISBN: 978-3-642-33160-2 (Print)ISBN: 978-3-642-33161-9 (Online)ISBN: 3-642-33160-2ISSN: 1865-0929pp.237-248