Cortically Inspired Sensor Fusion for Quadrotor Attitude...

Cortically Inspired Sensor Fusion

for Quadrotor Attitude Control

Florian Bergner

Abstract

Autonomous robotic systems, like quadrotors, need a fast, precise and reliable sensoryfeedback to stabilize motions in 3D space. Uncombined sensor signals cannot providesuch a feedback as they tend to be incomplete, inaccurate and noisy. For this reasonthere exist well established and successfully used algorithms which combine signals frommultiple sensors to the best possible sensory feedback. The process of combining multiplesensory signals to a more reliable and accurate result is called sensor fusion. State-of-the-art sensor fusion algorithms provide excellent results but often lack in �exibility androbustness.In this work I use a biologically plausible principle from neuroscience called populationcodes. I use a neural network of population codes to implement a sensor fusion algorithmfor quadrotor attitude estimation. I can show that such an implementation works withreal sensor data and delivers results which are comparable to those of state-of-the-artalgorithms. From this I deduce that population code based neural networks can possiblybe used in a generic way for the development of new, more �exible and robust sensorfusion algorithms.

Zusammenfassung

Autonome Systeme wie ein Quadrotor benötigen ein schnelles, präzises und zuverlässi-ges Feedbacksignal, um ihre Bewegungen im 3D-Raum stabilisieren zu können. EinzelneSensorsignale alleine können ein solches Feedbacksignal nicht bereitstellen, da sie oftunvollständig, ungenau und verrauscht sind. Aus diesem Grund werden mittels eta-blierter Algorithmen sehr erfolgreich Sensorsignale zu einem optimalen Feedbacksignalkombiniert. Das Kombinieren von Sensorsignalen zu einem zuverlässigeren und genaue-ren Ergebnis nennt man Sensor Fusion. Sensor Fusion Algorithmen, die dem aktuellenStand der Technik entsprechen, liefern exzellente Ergebnisse, sind aber oft nicht sehr�exibel und robust.In dieser Arbeit möchte ich biologisch plausible Prinzipien aus den Neurowissenschaftenverwenden. Ich verwende ein neuronales Netzwerk von Population Codes und konstru-iere einen Sensor Fusion Algorithmus, den ich zur Fluglageschätzung eines Quadrotorseinsetze. Ich zeige, dass ein solcher Algorithmus mit echten Sensordaten funktioniertund Ergebnisse liefert, die vergleichbar sind mit denen von etablierten Standard SensorFusion Algorithmen. Daraus schlieÿe ich, dass sich neuronale Netzwerke, die auf Popu-lation Codes basieren, möglicherweise auch für die Entwicklung neuer und robustererSensor Fusion Algorithmen eignen.

Inhaltsverzeichnis

1 Introduction 91.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91.2 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91.3 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2 Develop a quadrotor infrastructure for test �ights 112.1 The PX4 quadrotor (QR) . . . . . . . . . . . . . . . . . . . . . . . . . . 112.2 The optical tracking system . . . . . . . . . . . . . . . . . . . . . . . . . 122.3 The quadrotor infrastructure . . . . . . . . . . . . . . . . . . . . . . . . 13

3 Evaluate sensor data 153.1 Geometrical and physical relationships . . . . . . . . . . . . . . . . . . . 15

3.1.1 Transformation of coordinate systems . . . . . . . . . . . . . . . 153.1.2 Rotations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163.1.3 De�nition of object and world frames . . . . . . . . . . . . . . . . 173.1.4 From accelerations to angles . . . . . . . . . . . . . . . . . . . . . 183.1.5 From angular velocities to angles . . . . . . . . . . . . . . . . . . 193.1.6 From earth magnetic �eld to yaw angles . . . . . . . . . . . . . . 20

3.2 Data analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.2.1 Accelerometer data analysis . . . . . . . . . . . . . . . . . . . . . 213.2.2 Gyroscope data analysis . . . . . . . . . . . . . . . . . . . . . . . 273.2.3 Magnetometer data analysis . . . . . . . . . . . . . . . . . . . . . 283.2.4 Optical �ow sensor data analysis . . . . . . . . . . . . . . . . . . 293.2.5 Quadrotor online attitude estimator (EKF-QR) analysis . . . . . 31

4 Computing with population codes 334.1 Population codes (PC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

4.1.1 Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334.1.2 The preferred values of neurons in a population code . . . . . . . 344.1.3 The activation function of neurons in a population code . . . . . 354.1.4 Encode values in population codes . . . . . . . . . . . . . . . . . 364.1.5 Decode values of population code activity patterns . . . . . . . . 384.1.6 The scaling factor between di�erent activity patterns . . . . . . . 38

4.2 The multidimensional hidden layer network . . . . . . . . . . . . . . . . 394.2.1 Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394.2.2 The preferred values of hidden layer neurons . . . . . . . . . . . . 394.2.3 Activity mapping: From PC activities to HL activities . . . . . . 414.2.4 The activation function of hidden layer neurons . . . . . . . . . . 42

6 Inhaltsverzeichnis

4.2.5 Connections between hidden layer neurons . . . . . . . . . . . . . 42

4.2.6 Activity propagation in the hidden layer . . . . . . . . . . . . . . 45

4.2.7 Encode a D-dimensional value into a HL activity . . . . . . . . . 45

4.2.8 Decode hidden layer activities . . . . . . . . . . . . . . . . . . . . 46

4.3 Arbitrary functions in PC based neural networks (PCNN) . . . . . . . . 47

4.3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

4.3.2 Connections between hidden layer neurons and output layer neurons 48

4.3.3 From stabilized hidden layer activities to output sub-layer activities 49

4.4 Dynamic systems / Kalman Filters in PC based neural networks (PCNN) 49

4.4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

4.4.2 Connection weights between HL neurons for dynamic systems . . 50

4.4.3 Propagate the dynamic system in the hidden layer . . . . . . . . 51

4.4.4 Propagate the Kalman �lter in the hidden layer . . . . . . . . . . 52

5 State-of-the-art attitude estimators 55

5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

5.2 The generic Kalman Filter (KF) . . . . . . . . . . . . . . . . . . . . . . 57

5.3 The z-axis estimator (ZAE) . . . . . . . . . . . . . . . . . . . . . . . . . 59

5.4 The simpli�ed angle estimator (SAE) . . . . . . . . . . . . . . . . . . . . 63

6 Attitude estimators in PC based neural networks (PCNN) 67

6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

6.1.1 Deciding which attitude estimator to implement . . . . . . . . . . 67

6.1.2 Discussing the approach . . . . . . . . . . . . . . . . . . . . . . . 68

6.2 The preprocessing block . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

6.2.1 Topography of the neuronal network . . . . . . . . . . . . . . . . 69

6.2.2 Predict the rotated gravitational acceleration . . . . . . . . . . . 71

6.2.3 Calculate the accelerometer based feedback angles . . . . . . . . 73

6.3 The Kalman �lter block . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

6.3.1 Topography of the neural network . . . . . . . . . . . . . . . . . 74

6.3.2 Calculate recursive Kalman �lter estimations . . . . . . . . . . . 75

6.3.3 From Kalman gain to network parameters . . . . . . . . . . . . . 77

6.4 The linear acceleration estimation block . . . . . . . . . . . . . . . . . . 78

6.4.1 Topography of the neural network . . . . . . . . . . . . . . . . . 78

6.4.2 Update the rotated gravitational acceleration . . . . . . . . . . . 80

6.4.3 Calculate the linear acceleration . . . . . . . . . . . . . . . . . . 81

7 Evaluate and compare attitude estimators 83

7.1 The z-axis estimator (ZAE) . . . . . . . . . . . . . . . . . . . . . . . . . 83

7.1.1 Online Kalman gain calculation . . . . . . . . . . . . . . . . . . . 83

7.1.2 Performance of the estimator . . . . . . . . . . . . . . . . . . . . 84

7.2 The simpli�ed angle estimator (SAE) . . . . . . . . . . . . . . . . . . . . 86

7.3 The simpli�ed angle estimator in PCNNs (SAE-PC) . . . . . . . . . . . 88

7.3.1 Kalman gain and receptive �elds . . . . . . . . . . . . . . . . . . 88

7.3.2 Performance of the estimator . . . . . . . . . . . . . . . . . . . . 90

Inhaltsverzeichnis 7

8 Conclusions 95

Bibliography 98

1 Introduction

1.1 Motivation

Autonomous robotic systems, like quadrotors, need fast, precise and reliable sensoryfeedback to stabilize motions in 3D space. Uncombined sensor signals cannot be used togain a reliable sensory feedback as sensory signals tend to be incomplete, inaccurate andnoisy. For this reason the sensory feedback loop must combine signals of several sensorsand gain the best possible estimate for the system state. The process of combiningsensory signals to a more reliable and accurate result is called sensor fusion. State-of-the-art sensor fusion algorithms try to fuse sensory signals in a stochastic optimal wayby using internal models (Kalman �lter, extended Kalman �lter, Particle �lter). Another state-of-the-art approach for fusing signals is to make use of spectral di�erences insensory signals (Complementary �lter, Nonlinear complementary �lter) [14]�[16]. Thesealgorithms provide excellent results and have been used with great success in manyindustrial and commercial applications but often lack in �exibility and robustness. Oftenparameters have to be tuned for one speci�c setup.Biological systems need to fuse sensory information too and it is generally unknown howexactly cortically networks can do this. Nevertheless neuroscienti�c research providesprinciples on how it is believed that neurons process and fuse information. The basicprinciples are distributed computing, exchange of local information via feedforward andfeedback connections and using internal models [14]. Biological systems focus more onreliability and adaptability and not necessarily on optimality. Developing a sensor fusionalgorithm which makes use of these biologically plausible principles might result in asensor fusion algorithm which uses only simple operations but which is more robust,more generic and more �exible.The goal of this work is to �nd a �exible and generic, biologically plausible and corticallyinspired fusion model for quadrotor attitude control. The fusion model should aim foran implementation in low power embedded systems and thus use only simple operations.

1.2 Approach

In this work I want to �nd and implement a fusion model for quadrotor attitude estima-tion which uses biologically plausible principles and which thus becomes possibly moregeneric, robust and �exible. For the development and implementation of such a new fusi-on algorithm I want to make use of recent neuroscienti�c research results. [6]�[9] proposea biologically plausible principle on how recurrent cortical networks might represent andprocess information in populations of neurons. The principle is called population codingand a population of neurons is called population code (PC). Neural networks of such

10 1 Introduction

population codes (PCNN) are able to implement nonlinear functions, dynamic systemsor even Kalman �lters [6], [10]�[13]. So far PCNNs have mainly been used in neuros-cienti�c research projects but encoding information in populations of neurons seems alsoquite promising for engineering applications. By using PCNNs very complex models canbe implemented in a distributed way such that only simple operations like multiplica-tions and additions are needed. However it is not clear whether PCNNs will also workin real world applications with noisy sensor data. In this work I want to show that Ican use PCNNs to implement a Kalman �lter based fusion algorithm and thus estimatequadrotor attitudes by fusing sensory data in PCNNs. If I can show that such an im-plementation in PCNNs works then PCNNs o�er a very generic principle for the designof more sophisticated sensor fusion algorithms. After the design and implementation ofthe proposed algorithm I evaluate its performance by comparing it to state-of-the-artattitude estimators.

1.3 Overview

Before I can start with the implementation of the sensor fusion algorithm in PCNNsI need to do several things. First I need to prepare a quadrotor infrastructure whichprovides sensory data of a �ying quadrotor and ground truth reference data of an opticaltracking system. The infrastructure is developed and explained in chapter 2.Second I need to �nd and introduce geometrical and physical relationships for sensordata preprocessing. I do this in chapter 3 before I analyze and evaluate sensor dataof the quadrotor. The sensor data analysis is important because I have to make surethat the quadrotor infrastructure is working correctly. Also I have to get an impressionof how good the sensors are and where the sensors might have problems. In chapter4 I introduce and explain population codes and PCNNs. I collect ideas and principlesfrom multiple sources and create a consistent and generic mathematical framework forpopulation code and PCNN implementations.I give an overview of state-of-the-art fusion models in chapter 5 and I explain the Kalman�lter in more detail. Also I develop a modi�ed version of a Kalman �lter based state-of-the-art attitude estimator which I optimize for an implementation in PCNNs. Inchapter 6 I combine the results of chapter 4 and chapter 5 and explain in detail how Iimplement the modi�ed attitude estimator into PCNNs. Finally in chapter 7 I evaluateand compare the performance of the PCNN attitude estimator to state-of-the-art sensorfusion algorithms.

2 Develop a quadrotor infrastructure fortest �ights

We use a modi�ed PX4 quadrotor in combination with an optical tracking system toperform quadrotor test �ights. To work with such a quadrotor infrastructure we need tocontrol the quadrotor, we need to acquire �ight data of quadrotor sensors and we needto get reference data (ground truth data) of the tracking system. All this data has tobe synchronized and logged so that it can be used to test and evaluate newly developedalgorithms.The PX4 quadrotor and the tracking system cannot be used out of the box. The qua-drotor must be modi�ed and tuned and I need to develop software tools. Before we talkabout the details I will give you a brief overview of the systems we use.

2.1 The PX4 quadrotor (QR)

PX4FMU

PX4IOAR

PX4FLOW

PX4 quadrotor

QGroundconrol

flight base Linux PC

MAVLINK

radio controlPPM signal

Abbildung 2.1: Overview of PX4 quadrotor open source system

The PX4 quadrotor is a micro-air-vehicle (MAV) which consists of a PARROT ARDronequadrotor airframe, a PX4FMU �ight management unit, a PX4IOAR driver board forPARROT motors and a PX4FLOW optic �ow sensor. Besides the PARROT parts thePX4 quadrotor is part of the PX4 autopilot project, an open-source and open-hardwareproject which provides systems and solutions for academic, hobby and industrial use [1].The �ight management unit (FMU) is the core part of the PX4 quadrotor as it collects,processes and fuses available sensor data and uses the estimated attitude to stabilize the

12 2 Develop a quadrotor infrastructure for test �ights

attitude of the quadrotor. The FMU also receives control commands and can communi-cate with a �ight base on the ground. The FMU is powered by a powerful Cortex M4FARM microcontroller (168 MHz, 1 MB Flash, 192 KB SRAM) and also has an inertialmeasurement unit (IMU) on board. The IMU sensors are a 3D accelerometer (MPU-6000, 500 Hz), a 3D gyroscope (L3GD20, 500 Hz) and a 3D magnetometer (HMC5883L,100 Hz).The motor driver board PX4IOAR and the optic �ow sensor module PX4FLOW areconnected to the FMU. The optic �ow sensor module uses a CMOS camera, a sonar anda gyroscope to provide the FMU with rotation compensated ground speed and heightabove ground estimates.Per default the FMU and its well maintained open-source �rmware awaits control si-gnals in the form of pulse-position modulated signals (PPM signals) generated by astandard RC radio receiver. The FMU is also designed to communicate with the �ightbase via a wireless XBee connection. The �ight base is a standard computer which runsthe QGroundcontrol program [2] which acts for the PX4 quadrotor as open-sourceinformation and calibration center. The communication between the PX4 quadrotor andthe QGroundcontrol program makes use of a special communication protocol calledthe MAVLINK protocol [3]. The MAVLINK protocol is a quasi-standard for the com-munication between a �ying object and its �ight base.The QGroundcontrol program provides utilities to calibrate the sensors and the con-troller of the PX4 quadrotor. However the program cannot be used to control the PX4quadrotor using a joystick or other means. So the PX4 quadrotor and its �rmware needto be modi�ed in such away that we can control it by just using a PC and joystick. Wealso want to log �ight data of the PX4 quadrotor in such a way that we can easily readand work with it using MATLAB.

2.2 The optical tracking system

OptiTrack CAMs Motive

tracker PC

GT dataUDP multicast

Abbildung 2.2: Overview of natural point tracking system

The optical tracking system which is installed in the �ight lab of the NST is producedby NaturalPoint Inc. [4]. The tracking system uses 8 OptiTrack cameras and a programcalled Motive which tracks absolute positions of IR re�ecting markers with a samplerate of 120 Hz. The Motive program is running on a powerful, stationary PC in the�ight lab and we will call this PC from now on simply tracker PC.The Motive program enables us to calibrate the tracking system, to log sessions, tobroadcast tracking data and to bind markers to rigid bodies. Whenever we bind severalmarkers to a rigid body the Motive program not only tracks the absolute position of

2.3 The quadrotor infrastructure 13

the markers but also the orientation (roll, pitch and yaw angles) of the rigid body. Soin order to gain the absolute position and orientation of the quadrotor as ground truthdata (GT) from the tacking system, we place 3 markers onto the PX4 quadrotor andcombine the markers to a rigid body.The Motive program broadcasts ground truth data via a multicast UDP connectionand uses a special dynamic packet protocol. Unfortunately no Linux program or libraryexists which can receive and parse packets from theMotive program. So I'll use samplesource code of an exemplary packet parser, listen for packets using the Wireshark

program [5] and develop my own dynamic packet parser.

2.3 The quadrotor infrastructure

tflog


PX4 quadrotor

Motive

tracker PC

joystick

px4-lib


QGroundcontrol


MAVLINK

GT dataUDP multicast MAVLINK

log data

Abbildung 2.3: Overview of quadrotor infrastructure

In the previous the two sections we have seen that I have to make some modi�cationsand develop some programs. In the �rst step I modify the �rmware of the PX4 quadro-tor so that the quadrotor can receive control commands via the MAVLINK connection.As I want to use a standard NST UART-to-WiFi module for the wireless link betweenquadrotor and the �ight base PC I recon�gure the quadrotor �rmware so that the qua-drotor sends and receives MAVLINK messages over a high speed UART interface with4 MBaud. Nevertheless the QGroundcontrol still cannot send control commands tothe quadrotor or log �ight data.To solve this problem I develop a new program called t�og (test-�ight-logging program)which sits between the WiFi connection to the quadrotor and the QGroundcontrolprogram. The t�og program relays MAVLINK packets to the QGroundcontrol pro-gram so we can still use the QGroundcontrol program to display �ght data in realtime or to calibrate the PX4 quadrotor. The t�og program parses MAVLINK packagesusing an open-source MAVLINK library thus the t�og program gains access to MAV-LINK packages. As a result the t�og can log �ight data and send control commands

14 2 Develop a quadrotor infrastructure for test �ights

to the quadrotor. The t�og program recognizes joysticks and we can use a joystick tocontrol the quadrotor.If the tracker PC is connected to the �ight base PC then the t�og program parsestracker packages and logs ground truth data, too. As the t�og program bundles manyuseful utilities which might be used in other setups and projects I export these utilitiesinto a C-library called t�og-lib. This allows us to easily reuse solutions in other projectsand programs.Now that I have a program with creates log �les for �ight data of the quadrotor andground truth data I need a toolset which enables us to work with the gained data. The-refore I create a MATLAB library called px4-lib which imports log data into MATLAB,which synchronizes times stamps, which organizes data into data structures and whichprovides prede�ned plotting functions.

3 Evaluate sensor data

Before we can start to think about sensor fusion algorithms and their evaluation usingreal sensor data of quadrotor test �ights we have to know more about the sensor data weplan to use. Therefore we analyze and evaluate the sensor data. For sensor data analysiswe need to know something about coordinate systems, about rotations, about geometryand about physical relationships. So let's have a brief look at the mathematical andphysical tools we need and then we'll discuss and evaluate sensory data of the di�erentmeasurements systems.

3.1 Geometrical and physical relationships

3.1.1 Transformation of coordinate systems

When a quadrotor is moving in an environment we have at least two coordinate systems� one object coordinate system for the quadrotor and one �xed reference coordinate forthe environment. For simplicity we call the reference coordinate system world frame WFand the object coordinate system object frame OF. We are interested in the relationshipbetween the two coordinate systems and in how we can transform coordinates from onecoordinate system to the other one. For attitude control we are mainly interested inrotations between the world frame WF and the object frame OF and translations don'treally matter. Consequently don't need homogeneous coordinates and transformationswill reduce to rotation. In the following I want to introduce some naming conventionswhich I will use in a consistent way through the whole work.Let's look at a vector Ax. The elements of this vector are coordinates with respect tocoordinate system A (or frame A) which has a basis AB. The basis of A AB is a setof basis vectors bA

x , bAy , bA

z and these basis vectors span the space of the coordinatesystem A. If the basis vectors have a length of 1 and are orthogonal to each other thenwe call the coordinate system A a Cartesian coordinate system. Vectors x without anyspeci�ed coordinate system belong to the standard Cartesian coordinate system with abasis

B =(ex ey ez

)= I . (3.1)

Each axis of a coordinate system is orientated along a basis vector and each basis vectoris speci�ed with respect to the standard Cartesian coordinate system. The standardCartesian coordinates x of Ax can be calculated by a weighted linear sum of the basisvectors of frame A while the weights are the elements of Ax.

x = [Ax]1 · bAx + [Ax]2 · bA

y + [Ax]3 · bAz (3.2)

16 3 Evaluate sensor data

Note that I introduce here a special non-standard index operator [ · ]i which selects thei-th element of the vector the operator encloses.How can we transform coordinates between di�erent frames A and B? This can be doneby introducing a transformation matrix T. For example A

BT transforms vectors Bx offrame B to vectors Ax of frame A.

Ax = ABT · Bx (3.3)

We can determine the transformation matrix in the following way. Each column of thetransformation matrix A

BT must contain a basis vector AbB of frame B in coordinatesof frame A.

ABT =

(AbB

x AbBy AbB

z

)(3.4)

As a result for a Bx = ex which lies on the x-axis of frame B we get

AbBx = A

BT · ex . (3.5)

This is exactly the behavior we expect of ABT. If T belongs to the orthogonal group

O(n) then the inverse transformation T−1 is just the transposed matrix TT . The sameis true if T belongs to the special orthogonal group SO(n).

3.1.2 Rotations

Rotations around an arbitrary axis in a 3D space can be formalized in the two di�erentways. If we use the intrinsic way then each sub-rotation around a coordinate axis isde�ned with respect to the coordinate system we had before the sub-rotation and onlythe �rst sub-rotation is de�ned with respect to an axis of the initial coordinate system.If we use the extrinsic way then each sub-rotation is de�ned with respect to an axis ofthe initial coordinate system. As we are only interested in rotations of the quadrotorcoordinate system with respect to axes of the initial coordinate system, we use theextrinsic rotation formalism. The standard for rotation angles of nautical objects withrespect to a world frame are the so called Tait-Bryan angles. There are six possiblerepresentations for the Tait-Bryan angles as the there are multiple ways to change theorder of elemental rotations. For this work I use the standard axes convention (DIN9300) for airborne nautical vehicles which is the z-y-x convention. The nautical angles(cardan angles) are the roll φ, pitch θ and yaw ψ angles (RPY). The rotation matrixR(φ, θ, ψ) from the initial object frame to the current object frame for a given set of

RPY angels αRPY =(φ θ ψ

)Tis

R = R(φ, θ, ψ) = R(αRPY) = Rz(ψ) ·Ry(θ) ·Rx(φ) . (3.6)

The rotation matrices Rx(φ), Ry(θ) and Rz(ψ) describe elemental rotations around thespeci�ed axes x, y and z.

Rx(φ) =

1 0 00 cos(φ) − sin(φ)0 sin(φ) cos(φ)

(3.7)

3.1 Geometrical and physical relationships 17

Ry(θ) =

cos(θ) 0 sin(θ)0 1 0

− sin(θ) 0 cos(θ)

(3.8)

Rz(ψ) =

cos(ψ) − sin(ψ) 0sin(ψ) cos(ψ) 0

0 0 1

(3.9)

Rotation matrices belong to the special orthogonal group SO(n) thus the inverse rotationmatrix is

R−1 = RT (3.10)

R−1 = RTx (φ) ·RT

y (θ) ·RTz (ψ) (3.11)

R−1 = Rx(−φ) ·Ry(−θ) ·Rz(−ψ) . (3.12)

3.1.3 De�nition of object and world frames

xk

yk

zk

OFk

nose

right side

x

y

z

WFx0

y0

z0

OF0

Ground plane

Abbildung 3.1: The initial object frame OF0, the world frame WF and the object frameOFk of the quadrotor at time step k

In �gure 3.1 we see the di�erent coordinate systems I'll use in this work. The initialobject frame OF0 is the object frame of the quadrotor at time step k = 0 and usesstandard Cartesian coordinates with an identity matrix as basis. We can see that OF0

is identical to the world frame WF. The object frame OFk of the quadrotor at time stepk equals to the initial object frame OF0 rotated by the roll, pitch and yaw angles αRPY.So if x0 = 0x

0, y0 = 0y0 and z0 = 0z

0 are the unity vectors and axes of the initialobject frame OF0 and if we rotate these axes using the rotation matrix R(φk, θk, ψk)then we get the unity vectors xk = 0x

k, yk = 0yk and zk = 0z

k � the axes of objectframe OFk with respect to object frame OF0:(

xk yk zk)

= R(φk, θk, ψk) ·(x0 y0 z0

)(3.13)


If we want to transform coordinates of a vector x = 0x from OF0 to OFk we use thetransform matrix k

0T and get the coordinates kx of x in OFk:

kx = k0T · 0x (3.14)

kx = RT (φk, θk, ψk) · 0x (3.15)

The inverse coordinate transformation from OFk to OF0 is

0x = 0kT · kx (3.16)

0x = R(φk, θk, ψk) · kx . (3.17)

So we summarize that coordinate transformations between OF0 and OFk are equal tothe RPY rotation matrix R and its inverse:

0kT = R(φk, θk, ψk) (3.18)

k0T = RT (φk, θk, ψk) (3.19)

3.1.4 From accelerations to angles

y0

z0

g

yk

zk

φk

(a)

x0

z0

g

xk

zk

θk

(b)

Abbildung 3.2: Di�erent coordinate system views for rotations around the x-axis (roll,φk) and rotations around the y-axis (pitch, θk); in �gure (a) we look atthe x-axis which is perpendicular to the plane of this sheet of paper anddirectly points into our face; in �gure (b) we look at the y-axis whichis also pointing out of this sheet of paper; rotations around the z-axis(yaw angles, ψ) have no e�ect on the measurement of g

How can we use data of the accelerometer to estimate roll and pitch angles αRP? To �ndan answer to this question let's assume that the accelerometer only measures gravitatio-nal accelerations g. We will see later in subsection 3.2.1 that in general this assumption

3.1 Geometrical and physical relationships 19

is not true.When the quadrotor (see �gure 3.2) is in OF0 then g is parallel to the z-axis z0 of OF0:

g = g · z0 (3.20)

If the quadrotor is in object fame OFk which is rotated around the x-axis of objectframe OF0 (see �gure 3.2a) by a roll angle φk or around the y-axis (see �gure 3.2b) by apitch angle θk then the acceleration sensor onboard of the quadrotor measures a rotatedgravitational acceleration arot. Rotations around the z-axis of object frame OF0 (yawangles ψ) have no e�ect on arot and thus arot cannot be used to determine yaw anglesψ.We measure a rotated gravitational acceleration arot because g is still parallel to z0 butthe sensor measures g now in OFk which is rotated against OF0. arot is only equal to gwhen OFk is not rotated against OF0 and φk = θk = 0. In general arot is parallel to thez-axis of OF0 with respect to OFk:

akrot = g · kz0 (3.21)

We can use this relationship in combination with some geometrical re�ections to calcu-late roll and pitch angles

φk = arctan2([kz

0]2,[kz

0]3

)(3.22)

θk = −arctan2([kz

0]1,[kz

0]3

)(3.23)

or equally

φk = arctan2([

akrot

]2,[akrot

]3

)(3.24)

θk = −arctan2([

akrot

]1,[akrot

]3

). (3.25)

3.1.5 From angular velocities to angles

The gyroscope onboard the quadrotor measures angular velocities αRPY =(φ θ ψ

)T.

We can integrate these angular velocities and get angles αRPY:

αRPY(t) =

∫ t

t=0αRPY(t) dt (3.26)

We can integrate αRPY also numerically using a simple sum approximation (explicitEuler method with step width dt):

αkRPY = αk−1RPY + αkRPY · dt (3.27)

(3.28)


3.1.6 From earth magnetic �eld to yaw angles

x0

North

z0b

(a)

Northx0

y0

xk

yk

ψk

b

(b)

Abbildung 3.3: Di�erent views on the initial object frame OF0; the xy-plane of OF0 isparallel to the ground; �gure (a) is a side view and the x-axis x0 pointsto the north; the earth magnetic �eld vector b points to the ground andto the north; �gure (b) is a top view; b is the projection of b into theground plane; the object frame OFk is rotated around the z-axis (yaw,ψk)

The magnetometer of the quadrotor measures the earth magnetic �eld b. Magnetic �eldvectors always point from the magnetic north pole to the magnetic south pole. So inthe northern hemisphere b points to the ground and to the north because the magneticsouth pole is located near the geological north pole of the earth (see �gure 3.3a). Thusthe projection b of the earth magnetic �eld b to the ground plane (xy-plane) points tothe north (see �gure 3.3b). We see in �gure 3.3b that the projection b is always parallelto the x-axis x0 of the object frame OF0:

b = [b]1 · x0 (3.29)

The magnetometer measures b always with respect to the object frame OFk. So wemeasure a rotated earth magnetic �eld vector bkrot. As longs as the roll and pitch anglesare known we can use bkrot to calculate the yaw angle ψk. Therefore we have to transformbkrot partially back to OF0 such that the z-axes of OFk and OF0 are identical (see �gure3.3b). We call this intermediate object frame OFk and bkrot is the rotated earth magnetic�eld with respect to OFk:

bkrot = R(φk, θk, 0) · bkrot (3.30)

ˆbkrot is equal to b with respect to object frame OFk and is consequently parallel to thex-axis kx

0 of OF0 with respect to object frame OFk:

ˆbkrot = [b]1 · kx0 (3.31)

This relationship in combination with geometrical re�ections leads to the yaw angle

ψk = −arctan2([

ˆbkrot

]2,[ˆbkrot

]1

). (3.32)

3.2 Data analysis 21

3.2 Data analysis

For the data analysis I use log data of one quadrotor test �ight. The log data is generatedby the t�og program and contains �ight data of the quadrotor and ground truth dataof the tracker. The log data is imported to MATLAB using the px4-lib MATLABlibrary. The data is organized into data structures. Di�erent time stamps generatedby di�erent systems (quadrotor, tracker PC and �ight base PC) are synchronized andsensory data of di�erent sources with di�erent coordinate systems is transformed to onecommon coordinate system. As the di�erent data sources use di�erent sample rates Iinterpolate all the data to one common sample rate. In combination with the geometricaland physical relationships (see section 3.1) we can now analyze and evaluate step bystep the di�erent data sources and sensors.

3.2.1 Accelerometer data analysis

The accerlerometer on the quadrotor measures accelerations with respect to its currentobject frame OFk. As we generally cannot neglect linear accelerations the accelerometeralways measures a combination of linear accelerations aklin and rotated gravitationalaccelerations akrot. This combination of accelerations aknet is usually called net linearacceleration

aknet = aklin + akrot . (3.33)

The net linear acceleration aknet cannot be easily separated into its components aklinand akrot. However we can argue that the quadrotor is only translating very slowly andsmoothly so we consequently can make the assumption that aklin ≈ 0 and thus

akrot ≈ aknet . (3.34)

Under this assumption we can calculate the roll and pitch angles φk and θk using eq. 3.24and 3.25 and get results which we can see in �gure 3.4a and 3.4b. Clearly calculatingroll and pitch angles this way is not good at all. The results show only a very poorperformance and are just not usable. From this we deduce that the assumption aklin ≈ 0is wrong.

acc.

GT

φis

rollangle

inrad

t in sec.

roll angle φ

0 10 20 30 40 50 60 70

−0.2

−0.1

0

0.1

0.2

(a)


acc.

GTθis

pitch

angle

inrad

t in sec.

pitch angle θ

0 10 20 30 40 50 60 70

−0.2

−0.1

0

0.1

0.2

(b)

Abbildung 3.4: Calculated roll and pitch angles using only accelerometer data comparedto ground truth data (GT)

Let's use ground truth data of the tracking system and try to evaluate the performanceof the accelerometer in measuring acceleration. We can use RPY angles of the groundtruth data (GT) to calculate rotated gravitation accelerations akrot,GT with respect toobject frame OFk:

akrot,GT = RT (φkGT, θkGT, ψ

kGT) · g (3.35)

As we can see akrot,GT only bases on ground truth data. I assume that ground truth data

is very good and that I can use it as reference data so I can use akrot,GT as reference, too.In subsection 3.2.2 I will try to determine how good RPY angles of the ground truthdata really are.If we subtract akrot,GT from aknet we get a linear acceleration aklin, acc with respect toobject frame OFk:

aklin, acc = aknet − akrot,GT (3.36)

We can transform aklin, acc from the object frame OFk back to OF0 and compare 0aklin, acc

to linear accelerations 0aklin,GT:

0aklin, acc = R(φkGT, θ

kGT, ψ

kGT) · aklin, acc (3.37)

Linear accelerations 0aklin,GT are second derivatives of absolute quadrotor positions which

have been measured by the tracking system with sub-millimeter accuracy. So 0aklin,GT

are really good and can de�nitively be used as reference.Let's have a look at �gures 3.7a, 3.7b and 3.7c and compare 0a

klin,GT and 0a

klin, acc in

all the three dimensions. The �rst impression we get is that linear accelerations alin areindeed not too small and this delivers right away the answer why our previous assump-tion alin ≈ 0 is not ful�lled at all. We also see that 0a

klin,GT and 0a

klin, acc are su�ciently

close to each other with respect to the x and y dimensions. So with respect to these


dimension the accelerometer delivers reasonable results.However there is a signi�cant di�erence between 0a

klin,GT and 0a

klin, acc with respect to

the z dimension. Let's �nd an explanation to this. In general measurements of the ac-celerometer � we can see one in �gures 3.5a 3.5b and 3.5c � are quite noisy. The mainreason for this noise are vibrations generated by the motors of the quadrotor. As soonas we start the motors of the quadrotor at t ≈ 5 s we observe signi�cantly more noise in�gures 3.5a 3.5b and 3.5c. When the quadrotor starts to take o� at t ≈ 10 s we can seein �gure 3.5c that the z-axis acceleration starts to decrease vastly and the deviation of

0aklin, acc to the reference 0a

klin,GT increases (see �gure 3.7c).

In �gures 3.6a,3.6b and 3.6c we can see that the z component of akrot,GT is almost con-stant over time while the x and y components change over time.As a result changes in the z component of anet lead to changes in the z component of

0aklin, acc so that we can argue that deviations between 0a

klin, acc and 0a

klin,GT in the z

component are caused by inaccurate measurements of the accelerometer with respect tothe z-axis. The z-axis of the accelerometer is always parallel to the thrust axis of thequadrotor so one might assume that thrust changes in�uence accelerometer measure-ments in that axis in a negative way.We sum up that the accelerometer measures net linear accelerations, that linear accele-rations cannot be neglected, that the accelerometer measures accelerations in x and ydirection with su�cient accuracy and that the z component of the measured accelerationis erroneous.


filtered

raw

[anet] 1

isaccelerationin

m s2

t in sec.

net linear acceleration in x direction [anet]1

0 10 20 30 40 50 60 70−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

(a)

filtered

raw

[anet] 2

isaccelerationin

m s2

t in sec.

net linear acceleration in y direction [anet]2

0 10 20 30 40 50 60 70−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

(b)

filtered

raw

[anet] 3

isaccelerationin

m s2

t in sec.

net linear acceleration in z direction [anet]3

0 10 20 30 40 50 60 700

5

10

15

20

(c)

Abbildung 3.5: Raw and �ltered sensor data of the accelerometer in object frame OFk;the �ltered data was �ltered by a Butterworth �lter with a cuto� fre-quency of 3 Hz


GT RPY and g

[aro

t] 1

isaccelerationin

m s2

t in sec.

rotated gravitational acceleration in x direction [arot]1

0 10 20 30 40 50 60 70−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

(a)

GT RPY and g

[aro

t] 2

isaccelerationin

m s2

t in sec.

rotated gravitational acceleration in y direction [arot]2

0 10 20 30 40 50 60 70−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

(b)

GT RPY and g

[aro

t] 3

isaccelerationin

m s2

t in sec.

rotated gravitational acceleration in z direction [arot]3

0 10 20 30 40 50 60 700

2

4

6

8

10

12

(c)

Abbildung 3.6: Rotated gravitational acceleration arot which only bases on ground truthdata (GT) in object frame OFk


GT RPY and anet

GT

[ 0alin] 1

isaccelerationin

m s2

t in sec.

acceleration in x direction [0alin]1

0 10 20 30 40 50 60 70−1.5

−1

−0.5

0

0.5

1

1.5

2

(a)

GT RPY and anet

GT

[ 0alin] 2

isaccelerationin

m s2

t in sec.

acceleration in y direction [0alin]2

0 10 20 30 40 50 60 70−1.5

−1

−0.5

0

0.5

1

1.5

2

(b)

GT RPY and anet

GT

[ 0alin] 3

isaccelerationin

m s2

t in sec.

acceleration in z direction [0alin]3

0 10 20 30 40 50 60 70−10

−5

0

5

10

(c)

Abbildung 3.7: Comparison of linear accelerations 0alin in object frame OF0; blue is

0alin,GT which bases only on ground truth data and red is 0alin, acc whichbases on anet and RPY angles of the ground truth data; anet is measuredby the accelerometer


3.2.2 Gyroscope data analysis

The gyroscope onboard the quadrotor measures angular velocities αRPY. We can inte-grate the these angular velocities (see eq. 3.27) and get the RPY angles αRPY, gyro. Thegyroscope is the best sensor to measure RPY angles as it measures angular velocitieswith high precision and reliability in a direct way. Nevertheless the angles αRPY, gyro

tend to drift over time as we can see �gures in 3.8a, 3.8b and 3.8c. The drift is partiallycaused by the properties of the gyroscope sensor and partially cause by the integrationof noise and measurement errors. If we have a closer look at �gure 3.8b we notice thatthere is some di�erence between the pitch angles θ from t = 18 s to t = 32 s. If weconsider that the gyroscope is very reliable and accurate beside the drift overtime wecan conclude that the tracker based pitch angle is erroneous from t = 18 s to t = 32 s. Areason could be that the tracking system lost a marker because the marker was partiallyoccluded. This can happen because the camera arrangement for the tracking system isnot ideal to track marker in all directions with equal precision and coverage.We sum up that the gyroscope is the best sensor for estimating RPY angles, that RPYangles based on gyroscope data have some drift over time and that the tracking sys-tem estimates RPY angles accurately most of the time but sometimes looses a trackedmarker and thus delivers erroneous results.

gyro.

GT

φis

rollangle

inrad

t in sec.

roll angle φ = [αRPY]1

0 10 20 30 40 50 60 70

−0.2

−0.1

0

0.1

0.2

(a)


gyro.

GT

θis

pitch

angle

inrad

t in sec.

pitch angle θ = [αRPY]2

0 10 20 30 40 50 60 70

−0.2

−0.1

0

0.1

0.2

(b)

gyro.

GT

ψisyaw

angle

inrad

t in sec.

yaw angle ψ = [αRPY]3

0 10 20 30 40 50 60 70−0.5

0

0.5

1

(c)

Abbildung 3.8: Comparison of roll, pitch and yaw angles which base on gyroscope mea-surements or on ground truth data (GT); we deduced in subsection 3.2.5that the ground truth system reports erroneous pitch angles from t = 18 sto t = 32 s

3.2.3 Magnetometer data analysis

The magnetometer onboard the quadrotor measures the earth magnetic �eld b. As wehave seen in subsection 3.1.6 we can use b to estimate yaw angles ψmag which base onmagnetometer measurements but also on roll and pitch angles of a di�erent data source.Let's have a look at �gure 3.9 where we can compare ψmag to yaw angles ψGT which baseon ground truth data of the tracking system. We observe that ψmag is not close to ψGT

but has a more or less constant o�set to ψGT. The o�set occurs as soon as the motorsof the quadrotor are started at t ≈ 5 s. Form this circumstance one can deduce that


high currents in�uence the magnetic �eld around the quadrotor and the magnetometermeasures b with some errors. Consequently we notice that the magnetometer is onlyuseful if the non-constant o�set in ψmag can somehow be canceled out.

mag. and RP of GT

GT

ψisyaw

angle

inrad

t in sec.

yaw angle ψ

0 10 20 30 40 50 60 70−0.5

0

0.5

1

Abbildung 3.9: Comparison of yaw angles which base on ground truth data (GT) orwhich base on magnetometer measurements; when the motors of thequadrotor are turned on at t ≈ 5 s we observe an o�set in ψmag

3.2.4 Optical �ow sensor data analysis

The quadrotor has also a optic �ow sensor module installed which measures RPY com-pensated ground speeds and the height above ground. The optic �ow sensor uses aCMOS camera to recognize the �ow of detected feature points on the ground and withthat the ground speed of the drone. The sonar of the optic �ow sensor module measuresthe height above ground from a minimum height of h ≈ 30 cm to a maximum height ofh ≈ 4.5 m.In �gure 3.10 we compare the ground speeds vx and vy and the height above groundh to ground truth data. We observe that the optic �ow sensor delivers reasonable butnot always very reliable results. The peaks in �gure 3.10c are time instances when thesonar missed an echo and thus assumed maximum height above ground. Experimentshave shown that the optic �ow module must be placed somewhere near the center of thequadrotor so that rotor noise and thrust are avoided. These two e�ects greatly disturbsonar measurements and render them useless.So we sum up that the optic �ow module delivers information no other on-board sensorof the drone can deliver, that data of the optic �ow module is consistent to ground truthdata and that data of the optic �ow module is not reliable enough to use it without anyadditional feedback.


optical flow

GT

v xisvelocity

inm s

t in sec.

velocity in x direction vx

0 5 10 15 20 25 30 35 40 45 50−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

(a)

optical flow

GT

v yisvelocity

inm s

t in sec.

velocity in y direction vy

0 5 10 15 20 25 30 35 40 45 50−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

(b)

sonar

GT

hisheightin

meter

t in sec.

height h

0 5 10 15 20 25 30 35 40 45 500

1

2

3

4

5

(c)

Abbildung 3.10: Comparison of optic �ow sensor data to ground truth data; note thatwe use for this plot data of a di�erent test �ight consequently this plotcannot be compared to other plots of this chapter.


3.2.5 Quadrotor online attitude estimator (EKF-QR) analysis

The PX4 quadrotor uses an extended Kalman �lter (EKF) to estimate RPY anglesαRPY,EKF. The extended Kalman �lter fuses information of the accelerometer, the gy-roscope and the magnetometer. Let's compare the estimated angles αRPY,EKF to RPYangles of ground truth data and have a look at �gures 3.11a, 3.11b and 3.11c. We seein �gure 3.11a that roll angles φEKF are very close to roll angles φGT of the groundtruth data and that there is overall only a root mean squared error φerr, rms of about0.55 degrees between φEKF and φGT.Having a look at the pitch angles in �gure 3.11b, we observe that there is signi�cantdi�erence between θEKF and θGT from t = 18 s to t = 32 s. This is consistent to whatwe have observed in subsection 3.2.2 where we concluded that the ground truth systemreports erroneous pitch angles from t = 18 s to t = 32 s. Overall θEKF and θGT are closeto each other and an error of θerr, rms = 0.9439 is reasonable.Finally we examine the estimated yaw angles of the extended Kalman �lter in �gure3.11c and we observe again a signi�cant di�erence between ψEKF and ψGT. ψEKF isclose to ψmag while ψGT is close to ψgyro. From subsection 3.2.3 we know that ψmag iserroneous and cannot be relied upon on any time. So we conclude that for estimatingψ the extended Kalman �lter puts too much weight on the magnetometer and too fewweight on the gyroscope and thus delivers bad estimates for the yaw angle psi.We sum up that the extended Kalman �lter can estimate roll and pitch angles veryaccurately, that sometimes the pitch angle of the EKF is right while the pitch angle ofthe ground truth system is wrong and that yaw angles of the EKF are erroneous andhave some more or less constant o�set to real yaw angles of the quadrotor.

EKF-QR

GT

φis

rollangle

inrad

t in sec.

roll angle φ = [αRPY]1, φerr, rms = 0.5501 deg

0 10 20 30 40 50 60 70

−0.2

−0.1

0

0.1

0.2

(a)


EKF-QR

GT

θis

pitch

angle

inrad

t in sec.

pitch angle θ = [αRPY]2, θerr, rms = 0.9439 deg

0 10 20 30 40 50 60 70

−0.2

−0.1

0

0.1

0.2

(b)

mag. and RP of EKF

mag. and RP of GT

gyro.

EKF-QR

GT

ψisyaw

angle

inrad

t in sec.

yaw angle ψ = [αRPY]3

0 10 20 30 40 50 60 70−0.5

0

0.5

1

1.5

(c)

Abbildung 3.11: The roll, pitch and yaw angles αRPY,EKF of the extended Kalman �lterof the quadrotor (EKF-QR) in comparsion to RPY angles of othersources; we deduced in subsection 3.2.5 that the ground truth systemreports erroneous pitch angles from t = 18 s to t = 32 s

4 Computing with population codes

Neuroscienti�c research has deduced many biological plausible principles on how recur-rent cortical networks might represent information in populations of neurons [6]�[9], howthe cortex might realize spatial transformations using linear and nonlinear mappings [6],[10]�[12] and how recurrent neural networks might realize optimal sensor integration ordynamic system models [13]. So far these principles have mainly been used in neuros-cience to develop new models which are consistent with neurophysiological data andobservations. With the help of these models neuroscienti�c research tries to gain newinsights into how the brain might process information.The principle of representing information in population of neurons seems also quitepromising for engineering applications like sensor fusion. State-of-the-art algorithms forsensor fusion like Kalman �lters, complementary �lters and particle �lters are alreadywell established and used with great success in many industrial and commercial app-lications. These algorithms fuses information in an optimal manner but often lack in�exibility and robustness. Often parameters have to be tuned by hand for one speci�csetup. On the other hand neural networks process information in a decentralized way ina near optimal way which is often more robust.This chapter represents the very core of my work. I collect ideas and principles of multiplepapers [6]�[13] which deal with the principle of representing information in populationsof neurons and which introduce principles on how to calculate with these populations ofneurons in the fashion of recurrent cortical networks. I convert these ideas and princip-les into a generic mathematical representation which will enable us to use and test thecollected biologically inspired principles in real world applications. I implement all thesegeneric formulas in a hybrid MATLAB/C library pc-lib so that the collected principlescan easily be used. I use this library to validate that the collected principles not onlywork in theory but also in real world applications with real sensor data.So this chapter introduces the biologically inspired principles I plan to use and verify,puts everything into a generic and consistent mathematical representation and providesinsights which will be needed in the following chapters.

4.1 Population codes (PC)

The principle of encoding information in populations of neurons is called populationcoding. This principle is explained and investigated in [6]�[9].

4.1.1 Basics

A population code (PC) is a population of neurons whose collective activity patternencodes a scalar value. Each neuron of a population code has an unique preferred value.

34 4 Computing with population codes

The preferred value of a neuron represents the input the neuron likes most. Wheneverthe input matches the preferred value of a neuron then that neuron �res with its hig-hest possible output activity. The input of a neuron is the weighted linear sum of inputactivities. Input activities are output activities of other neurons to which the currentneuron has input connections. With an increasing distance between the input and thepreferred value of a neuron the output activity decreases. This behavior is described bythe activation function.The activation function is bell-shaped and de�nes the receptive �eld of a neuron. Whe-never the di�erence between the input of a neuron and its preferred value is bigger thenthe receptive �eld than the output activity of the neuron becomes zero.The neurons of a population code are ordered in a topographical manner. Neighboringneurons have preferred values which di�er only slightly and their receptive �elds overlap.In its simplest form a population code is just a row of neurons with di�erent preferredvalues and the population code activity pattern � which is just a set of output activitiesof population code neurons � encodes a scalar value.

4.1.2 The preferred values of neurons in a population code

Let's have a closer look at the preferred values of neurons in a population code and athow we can notate them in a mathematical way. A population code (PC) is a set ofneurons where each neuron i has its unique preferred value pcpi. If a population code hasN neurons then the preferred values of these neurons can be combined to a preferredvalue vector ppc:

ppc =

pcp1...

pcpN

∈ RN×1 with pcpi <pcpj ∀i < j (4.1)

As we can see ppc organizes the preferred values in an ordered manner. We start withthe smallest preferred value at i = 1 and stop with the largest preferred value at i = N .In this way we order the neurons in a topographic manner.To make life easier I introduce an index operator [ · ]i. If we apply this index operatoronto a vector then this operator selects the i-th element of this vector:

[ppc]i = pcpi (4.2)

We will see in a moment that this index operator can also be applied on matrices whereit has a related meaning.If we have a set of D population codes which encodes D-dimensional values x thenwe can organize the preferred value vectors dppc of these populations codes by using apreferred value matrix Ppc:

Ppc =(

1ppc . . . Nppc

)= (pcpij) ∈ RN×D (4.3)

The d-th element of x is encoded by the d-th population code and the preferred valuevector dppc of this population code can be found in the d-th column of the preferred

4.1 Population codes (PC) 35

value matrix Ppc. If we apply the previously introduced index operator [ · ]k (see eq. 4.2)on a matrix then we get the k-th column of this matrix:

[Ppc]k = kppc (4.4)

If we only want to select some elements of a matrix I will use the following notation:

[Ppc]k = (pcpij) i = 1 . . . Nj = k

(4.5)

The preferred values of neurons in a population code can be chosen in many ways. Ifnot otherwise mentioned we assume that the preferred values of a population code arelinearly spaced and have common limits:

ppc =(−1 −0.8 −0.6 . . . 0 . . . 0.6 0.8 1.0

)T(4.6)

4.1.3 The activation function of neurons in a population code

Each neuron in a population code uses a bell-shaped activation function fact(x). Thewidth of the bell de�nes the width of the receptive �eld (RF) of a neuron. We can usea Gaussian function (see eq. 4.7, [9]) or a combination of a cosine and an exponentialfunction (see eq. 4.8, [13]) as activation function:

fact(x) = KL exp

(− x2

2σ2

)(4.7)

fact(x) = KL exp (KW · [cos(x)− 1]) (4.8)

If not otherwise mentioned we always use the activation function of eq. 4.8. The shapeof an activation function is controlled by several parameters. KL is the scaling factorand de�nes the maximum possible output activity of a neuron. For example if KL = 1then the maximal output activity of a neuron using this activation function is 1. Theparameters KW and σ in�uence the width of the receptive �eld. The receptive �eldwidth RF of an activation function is

RFp(KW) = arccos

(1 +

ln(p)

KW

)(4.9)

RFp(σ) = σ√

ln (p−2) . (4.10)

The value p de�nes the relative activity threshold. A certain input x is within thereceptive �eld whenever the relative activity p = fact(x)

KLis above the relative threshold

p (see �gure 4.1).


RF10% of exp-cosRF10% of Gaussian

exp-cos

Gaussianf a

ct(x)

x

Activation functions

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Abbildung 4.1: Examples for activity functions of neurons in a PC; we can see a Gaussianactivation function with parameters KL = 1, σ = 0.25, RF10% = 0.5365and a combined exp-cos activation function with parameters KL = 1,KW = 100, RF10% = 0.2150

4.1.4 Encode values in population codes

Let's see how we encode scalar values x into a population code activity pattern. If weencode a value x into an population code activity pattern then we perform a one-to-many mapping 1→ N . We can use the activity function fact and calculate the activitypcai of a neuron i for a given scalar value x:

pcai = fact(x− pcpi) (4.11)

We can organize the activities of neurons in a population code in the same way as wehave organized the preferred values. We introduce apc, an activity vector for a populationcode which encodes the scalar value x:

apc =

pca1...

pcaN

∈ RN×1 (4.12)

The activity vector apc can also be called activity pattern of a population code and isagain bell shaped. Instead of calculating the activity for each neuron we can calculate

4.1 Population codes (PC) 37

apc directly

apc = encActpc(x) = f (x · 1N − ppc) (4.13)

with

f(x) =

fact(x1)...

fact(xN )

(4.14)

and a vector 1N ∈ RN×1 whose entries are all set to 1. If we want to encode a D-dimensional value x we have D activity patterns dapc for D population codes. Everyactivity pattern dapc encodes one coordinate of x and we can organize the D activitypatterns using an activity matrix Apc:

Apc =(

1apc . . . Napc

)= (pcaij) ∈ RN×D (4.15)


3apc

2apc

1apc

activity

preferred values

Activities in PCs

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Abbildung 4.2: Activity patterns 1apc,2apc and

3apc of a PC with N = 71 neurons forthree di�erent encoded values 1x = 0.4, 2x = −0.2 and 3x = −0.5; theactivation function parameters are KL = 1 and KW = 50; note thateach circle of an activity pattern represents the activity of one neuronin the PC

4.1.5 Decode values of population code activity patterns

How can we decode an encoded value x of a given activity pattern apc? The scalar valuex can be decoded by taking the weighted linear sum of all the preferred values of theused population code [13]:

x = dcdActpc (apc) =aTpc ppc∑i

pcai(4.16)

We observe that the activities of population code neurons act as weights for the weightedlinear sum. The weighted linear sum is normalized with the sum of all activities.

4.1.6 The scaling factor between di�erent activity patterns

Often we need to know scaling factors between two di�erent activity patterns. For ex-ample the scaling of activity patterns might change after performing some calculation.

4.2 The multidimensional hidden layer network 39

We can �nd the scaling factor of a given activity pattern with respect to a referenceactivity pattern by calculating the sums of activities:

scaleOfAct (a,aref) =

∑i ai∑

i aref,i=

KL

KL,ref(4.17)

The scaling factor is independent of the encoded value x and independent of the preferredvalues the population codes use. This formula also works with hidden layer activitieswhich are introduced in section 4.2.

4.2 The multidimensional hidden layer network

Whenever we need to calculate with more than one input value we have to use a hiddenlayer network (HL) which can encode multiple scalar values into one single hidden layeractivity pattern. The construction of hidden layers is introduced and explained in [6],[13].

4.2.1 Basics

If we want to calculate with multiple population codes then we have to transfer po-pulation code activity patterns to a common, higher dimensional activity space. Thisactivity space is realized in a hidden layer. The hidden layer is � like a population code� a population of neurons with di�erent preferred values and a hidden layer activitypattern encodes multidimensional values x. Hidden layer activity patterns represent abasis function set so any arbitrary function � which can be constructed of this basisfunction set � can be realized in a 3 layer population code network. The �rst layer isthe input layer with a population code for every component of x. The second layer isthe hidden layer which encodes x in a common activity space and the third layer is theoutput layer with a set of population codes. The population codes of the output layerhave weighted connections to the hidden layer. We will discuss the 3 layer network indetail in section 4.3.Lateral connections between hidden layer neurons can stabilize hidden layer activitiesor realize dynamic systems. In this section I describe how a hidden layer looks like,how population code activities are mapped to hidden layer activities and how we canstabilize and propagate hidden layer activities.

4.2.2 The preferred values of hidden layer neurons

To represent all the components of a D-dimensional value x, hidden layer neurons musthave D-dimensional preferred values. Each dimension of a hidden layer preferred valuecorresponds to a population code of the input layer. The hidden layer must realize allcombinations of preferred values of the D population codes and thus the hidden layermust have P = ND neurons with P di�erent preferred values.The question now is: How can we organize the preferred values of hidden layer neuronsfor a given set of D population codes? First we de�ne pThl as a preferred value of a hiddenlayer neuron and each element of pThl corresponds to a preferred value of one neuron of


one of the D population codes. The d-th element of pThl is equal to one preferred value ofa neuron in the d-th population code. We remember the d-th population code representsthe d-th dimension of value x which a hidden layer activity pattern will encode. So forpThl we get

pThl =([

1ppc

]i1

. . .[Dppc

]iD

)∈ R1×D . (4.18)[

dppc

]idis the preferred value of neuron id in the d-th population code. If we put the

indices id of a hidden layer neuron with the preferred value pThl into an index row vectoriThl:

iThl =(i1 . . . iD

)∈ R1×D (4.19)

then this index row vector de�nes which population code neurons are represented bythis hidden layer neuron. The collection of kiThl of all P hidden layer neurons is de�nedas the hidden layer index matrix Ihl:

Ihl =

1iThl...

P iThl

∈ RP×D (4.20)

Each row k of that matrix de�nes which combination of neurons of the D populationcodes is represented by the k-th hidden layer neuron. The preferred value of the hiddenlayer neuron k is de�ned by the preferred values of these population code neurons.The index matrix Ihl can have any arbitrary organization of indices but we will alwaysuse the following scheme (N = 4, D = 2, P = 16):

Ihl =

1 11 21 31 42 12 22 32 43 13 23 33 44 14 24 34 4

(4.21)

Let's organize the hidden layer preferred values pThl to a preferred value matrix Phl:

Phl =

1pThl...

PpThl

∈ RP×D (4.22)


The k-th row of that matrix is equal to the preferred value kphl of hidden layer neuron kand the elements of kphl relate to population code preferred values dppc in the followingway:

[kpThl

]d

=[dppc

]ihl(k, d)

∀d ∈ 1 . . . D (4.23)

Note that[kpThl

]dis the preferred value of the hidden layer neuron k in dimension d,

that[dppc

]lis the preferred value of population code neuron l in the d-th population

code and that l = ihl(k, d) is a function which �nds the population code neuron l for agiven hidden layer neuron k and a given dimension d:

ihl(k, d) = hlikd =[iThl(k)

]d

(4.24)

4.2.3 Activity mapping: From PC activities to HL activities

Here we have a look at how we can map individual PC activities apc to individual HLactivities ahl. For each individual HL activity we assume that we have currently nohidden layer activity and that all the other PC activities are set to zero. How we cancombine all the D individual HL activities to one common HL activity which encodesx will be discussed later in subsection 4.2.7.The output of a PC neuron is connected only to neurons in the hidden layer which havethe same preferred value in the relevant dimension as that PC neuron. So under thepreviously introduced assumption hidden layer neurons k with a speci�c preferred value[kpThl

]din the relevant dimension d have the same activity as the corresponding neurons

l in PC d:

[dahl

]k

=[dapc

]l⇐⇒

[kpThl

]d

=[dppc

]l

∀k ∈ 1 . . . P ∧ (4.25)

∀l ∈ 1 . . . N[dahl

]k

=[dapc

]ihl(k, d)

(4.26)

[dahl

]kis the activity of hidden layer neuron k while dahl ∈ RP×1 is the hidden layer

activity pattern which dimension d contributes.[dapc

]lis the activity of population

code neuron l in population code d while dapc is the population code activity patternof population code d.


2ahl

1ahl

dim 2 (input PC 2) dim 1 (input PC 1)

Activity in 2D hidden layer

−1

−0.5

0

0.5

1

−1

−0.5

0

0.5

10

0.2

0.4

0.6

0.8

1

Abbildung 4.3: Hidden layer activities 1ahl and2ahl for two given PC activities 1apc

and 2apc in two di�erent dimensions d = 1 and d = 2; the PC activitiesencode the values dcdActpc

(1apc

)= 0.4 and dcdActpc

(2apc

)= −0.2;

each PC has N = 71 neurons thus the 2-dimensional hidden layer hasP = 5041 neurons; the parameters of the PC activation functions areKL = 1 and KW = 50; each circle in the plot represents the activity ofone hidden layer neuron

4.2.4 The activation function of hidden layer neurons

The activation function for hidden layer neurons is am dimensional bell shaped functionwith m ≤ D and has its maximum at x = 0 [13]:

fact, hl(x) = KL exp

(KW ·

[m∑i=1

cos ([x]i)−m])

(4.27)

This function is used whenever we want to know how close a given value is to a preferredvalue kphl of hidden layer neuron k.

4.2.5 Connections between hidden layer neurons

Hidden layer neurons can have connections to other hidden layer neurons. These connec-tions are called lateral connections.


A hidden layer with P neurons has P 2 lateral connection weights hlwkl. The lateralconnection weight hlwkl is the connection weight from hidden layer neuron l to hiddenlayer neuron k. All the connection weights together form the hidden layer weight matrixWhl.

Whl =

1wT

hl...

PwThl

=(

hlwkl

)∈ RP×P (4.28)

Each row vector kwThl contains all the input connection weights of a hidden layer neuron

k. The lateral weights can be chosen in such a way that Whl stabilizes hidden layeractivity patterns or that Whl realizes an arbitrary dynamic system g(x).But how do we have to chose these weights in Whl so that we achieve a desired behaviorin the hidden layer for the activity propagation process? The dynamic system functionxk+1 = g(xk) de�nes the value xk+1 which an activity pattern ak+1

hl has to encode attime step k+ 1 under the condition that we have at time step k the activity pattern akhl

which encodes the value xk. So if we look at a hidden layer neuron i with a preferredvalue iphl close to xk+1 then its input hidden layer neurons j with preferred valuesjphl close to xk should get big connection weights. Therefore we calculate the di�erenced(i, j) between the result of g(jphl) for an input neuron j and the preferred value iphl

of a center neuron i:

d(i, j) = g(jphl)− iphl (4.29)

The closer g(jphl) gets toiphl the smaller d(i, j) becomes. The di�erence d(k, l) is now

applied to the activation function fact,hl(x) and we get the lateral connection weighthlwkl from input neuron l to center neuron k [13]:

hlwkl = fact, hl (d(k, l)) (4.30)

hlwkl = KL · exp

(KW

[D∑i=1

cos([

g(lphl)− kphl

]i

)−D

])(4.31)


whl for pThl = (0 0)

whl for pThl = (−0.4 0.4)

dim 2 (input PC 2) dim 1 (input PC 1)

Lateral weights in 2D hidden layer

−1

−0.5

0

0.5

1

−1−0.5

00.5

10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Abbildung 4.4: The distribution of lateral weights whl for two hidden layer neurons, onewith a preferred value pThl =

(−0.4 0.4

)and one with a preferred value

pThl =(0 0

); this hidden layer only stabilizes activities, so the dynamic

system function g(x) = x is an identity function; the population codeshave N = 71 neurons, the hidden layer P = 5041 neurons; the activati-on functions for population code neurons and hidden layer neurons allhave the same parameters KL = 1 and KW = 50; if we use a sparserepresentation for Whl then the compression rate for the setting aboveis 83%

As the weight matrix Whl =(

hlwkl)is a P × P matrix and as P grows exponentially

when we add new dimensions calculating these weights can take a lot of time and Whl

consumes a lot of memory space.Let's calculate an example so that you get a better impression of what I mean. Ifwe have three input dimensions D = 3 and N = 21 neurons per population codethen the hidden layer has P = 213 = 9261 neurons. Consequently the weight matrixWhl ∈ R9261×9261 has 92612 = 85 766 121 elements. As the elements are of type double

and as an element of type double needs sizeof(double) = 8 Bytes, the weight matrix willneed sizeof(Whl) = 655 MB of memory space.This explosion of occupied resources for increasing dimensions is called Curse of Dimen-sionality. There are several strategies with which we can reduce these e�ect.For an implementation we should use as few dimensions as possible. This can often be


achieved by applying the principle of divide-and-conquer which means that we shouldtry to reduce big, centralized and high dimensional problems to many small, distributedand low dimensional problems.Using a sparse representation for the weight matrix is bene�cial, too. As weight distri-butions are usually bell-shaped (see �gure 4.4) a lot of weight entries in Whl are closeto zero. So if we only store weights which are above a speci�ed threshold using a sparserepresentation for Whl we can save a lot of memory space. Compression rates of 80%to 90% are quite common and thus sizeof(Whl) ≈ 65 MB. A sparse weight matrix alsospeeds up processing as matrix-vector multiplications are much faster because the ma-trix entry count is reduced vastly to about 10% of the original entry count. The pc-libmakes use of sparse matrices and bene�ts.

4.2.6 Activity propagation in the hidden layer

How can we calculate the new hidden layer activity ak+1hl for a given activity akhl while

we consider only the current activity akhl and only lateral connections Whl? We have tocalculate the weighted squared sum

[akhl

]iof akhl for all the P hidden layer neurons i:

[akhl

]i

=([

WThl

]i· akhl

)2∀i ∈ 1 . . . P (4.32)

[akhl

]i

=(iwT

hl · akhl

)2∀i ∈ 1 . . . P (4.33)

[akhl

]i

=P∑j=1

(hlwij · hlakj

)2∀i ∈ 1 . . . P (4.34)

Then the hidden layer activity akhl is normalized using divisive normalization and we getthe propagated/stabilized new hidden layer activity ak+1

hl [13]:

[ak+1

hl

]i

=[h(akhl)

]i

=

[akhl

]i

µ+ η∑P

j=1

[akhl

]j

∀i ∈ 1 . . . P (4.35)

µ and η are tuning parameters. So we can propagate hidden layer activities using thehidden layer propagation function h(ahl):

ak+1hl = h(akhl) (4.36)

4.2.7 Encode a D-dimensional value into a HL activity

We have D activity patterns dapc which encode a D dimensional value x and we organizedapc by introducing a population code activity matrix Apc:

Apc =(

1apc . . . Dapc

)∈ RN×D (4.37)


As we discussed previously in subsection 4.2.3 we can calculate an individual hiddenlayer activity dahl for each population code activity dapc. All these individual hiddenlayer activities are organized in a hidden layer activity matrix Ahl:

Ahl =(

1ahl . . . Dahl

)∈ RP×D (4.38)

The question now is: How can we encode x to a single hidden layer activity ahl? So farwe have only encoded components of x into several individual hidden layer activitiesdahl.The answer is that we just have to sum up all the individual hidden layer activities dahl

which encode components of x [13]:

a0hl =

D∑d=1

[Ahl]d (4.39)

Then we stabilize the hidden layer activity using the activity propagation function h(ahl)in combination with a weight matrix Whl which implements g(x) = x [13]:

ak+1hl = h(akhl) (4.40)

Stabilizing activities means that the hidden layer activity ahl converges to a distinct bell-shaped pattern which accurately encodes x. This usually happens after 3 to 4 iterations.

ak=4hl

dim 2 dim 1

a0hl

dim 2 dim 1

Activity in 2D hidden layerActivity in 2D hidden layer

−1

0

1

−1

0

1

−1

0

1

−1

0

10

0.2

0.4

0.6

0.8

1

0

0.5

1

1.5

2

Abbildung 4.5: Stabilizing hidden layer activities: a0hl is the hidden layer activity after

summing up all the individual hidden layer activities dahl; ak=4hl is the

hidden layer activity after 4 iterations; the hidden layer activity clearly

stabilizes and encodes the value x =(0.4 −0.2

)T4.2.8 Decode hidden layer activities

A stabilized activity pattern in the hidden layer is a D dimensional bell-shaped surface(see �gure 4.5). The D dimensional value x which is encoded in a hidden layer activity

4.3 Arbitrary functions in PC based neural networks (PCNN) 47

pattern ahl can be decoded in the same manner as a population code activity pattern[13]:

x = dcdActhl (ahl) =aThl Phl∑Pi

hlai(4.41)

4.3 Arbitrary functions in PC based neural networks(PCNN)

We can use a 3 layer neural network which consists of an input layer (IL) and a hiddenlayer (HL) and an output layer (OL) to realize any arbitrary mathematical function aslong as the function is in the basis function set of the hidden layer. The principle of such3-layer networks is introduced in [6], [10]�[12].

4.3.1 Overview

PC1

PC2

PC3

1

2

3

IL1

1Whl

HL1 : g(x) = x

ak+1hl = h(ak

hl)PC1

1Wpc

SL1 : 1r(x)

PC22Wpc

SL2 : 2r(x)

PC33Wpc

SL3 : 3r(x)

OL1

dcdActpc(apc)

dcdActpc(apc)

dcdActpc(apc)

N × 1

1apc

N × 1

2apc

N × 1

3apc

P × 1ahl

P × 1ahl

P × 1ahl

N × 1

1apc, out

N × 1

2apc, out

N × 1

3apc, out

3 × 1x

3 × 1y

Abbildung 4.6: Example for a 3-layer population code based neural network; the networkconsists of an input layer (IL), a hidden layer (HL) and an output layer(OL) with separate sub-layers (SL) for each component of the outputvalue; the network implements an arbitrary function y = r(x)

We use population codes for the implementation of the 3 layer network. The input layerencodes the coordinates of a D dimensional value x using D population codes (see �gu-re 4.6). The D population code activity patterns dapc are mapped to the hidden layerwere hidden layer neurons encode x in a multidimensional activity pattern ahl. All thepossible hidden layer activity patterns together from a basis function set and the linearweighted sum of a basis function set can realize any arbitrary mathematical function �as long as the function is in the basis function set.The output layer consists of neurons which are connected to hidden layer neurons.Connection weights for connections from hidden layer neurons to output layer neuronsrealize the desired mathematical function which the 3-layer network has to implement.The mapping from hidden layer neurons to output layer neurons can take place in twodi�erent ways.Either we can use a multidimensional neural layer (like the one used in the hidden layer)


for the output layer so that we can implement functions which cannot be separated orwe can use Q population codes for Q output dimensions whenever we can separate thefunction into Q scalar function with a D-dimensional input. As long as it is possiblethe later form is preferred as it vastly reduces the amount of neurons we need for animplementation.If we can use population codes in the output layer we speak of sub-layers (SL) in theoutput layer. A sub-layer consists of an input weight matrix and a population code.The input weight matrix contains connection weights for connections from hidden layerneurons to neurons of the sub-layer population code. In this work we only discuss outputlayers which use population codes as this is the only form we need for our implementa-tions.

4.3.2 Connections between hidden layer neurons and output layer neurons

Connection weights pcwkl between P hidden layer neurons l and N population codeneurons k in the q-th population code of the output layer realize the function qr(x).A connection weight pcwkl de�nes the weight of a connection from hidden layer neuronl to population code neuron k. All these connection weights pcwkl together form theconnection weight matrix Wpc:

Wpc =

1wT

pc...

NwTpc

∈ RN×P (4.42)

Each row iwTpc of this matrix contains the input weights of one neuron i in a population

code of the output layer.How do we have to choose the connection weights between hidden layer neurons andneurons of the q-th population code so that they realize the function qr(x)? The functiony = qr(x) de�nes the value y the population code q has to encode in qapc when thehidden layer encodes the input value x in ahl. So if the preferred value lphl of a hiddenlayer input neuron l is close to x and if the preferred value [qppc]k of a population codeneuron is close to y then the connection weight pcwkl from neuron l to neuron k shouldbe big. We calculate the di�erence d(k, l) between qr(lphl) for an input neuron l andthe preferred value [qppc]k of a population code neuron k:

d(k, l) = qr(lphl)− [qppc]k (4.43)

When we apply this di�erence to the activation function fact, pc which is used by popu-lation code neurons then we get the desired connection weights

pcwkl = fact(d(k, l)) (4.44)

pcwkl = KL · exp(KW ·

[cos(qr(lphl)− [qppc]k

)− 1])

. (4.45)

Note that we need a weight matrix Wpc = (pcwkl) for each population code q of the Qpopulation codes in the output layer and that a combination of the q-th weight matrixqWpc and the q-th population code form the q-th sub-layer.


4.3.3 From stabilized hidden layer activities to output sub-layer activities

We have discussed in subsection 4.3.2 how we determine connection weights from hid-den layer neurons to population code neurons such that these connections realize thedesired input-output mapping y = r(x). These connection weights in combination witha stabilized hidden layer activity ahl enable us to calculate output activities qapc, out.Let's have a look at sub-layer SLq with a weight matrix qWpc and population code PCq.For a given hidden layer activity ahl and a weight matrix qWpc we can determine theactivity qapc, out of population code PCq in sub-layer SLq:

qapc, out = qWpc · ahl (4.46)

We can do this for all the Q sub-layers in the output layer and get an output activitymatrix Apc, out:

Apc, out =(

1apc, out . . . Qapc, out

)∈ RN×Q (4.47)

So, overall this 3 layer neural network has D population codes in the input layer and Qsub-layers in the output layer and the network implements a function y = r(x):

r : RD×1 → RQ×1 (4.48)

r : x→ y = r(x) (4.49)

r(x) =

1r(x)...

Qr(x)

(4.50)

4.4 Dynamic systems / Kalman Filters in PC based neuralnetworks (PCNN)

We can tune lateral connections in the hidden layer in such a way that the hidden layercan implement a dynamic system or even a Kalman �lter. Paper [13] introduces theprinciple and proposes an implementation scheme.

4.4.1 Overview

We have already mentioned in section 4.2 that we can tune lateral connection weightsin the hidden layer in such away that a propagation of a hidden layer activity followsthe rules of a dynamic system.To realize a dynamic system in a hidden layer we need to divide the hidden layer spacewith D = S + C dimensions into a system state subspace with S dimensions and acontrol input subspace with C dimensions. In such a con�guration population codes ofthe input layer are only connected to hidden layer neurons of the control input subspace.However if we want to realize a Kalman �lter then some population codes of the inputlayer are also connected to hidden layer neurons of the system state subspace. These


population codes feed feedback values into the system state subspace and the populationcode neurons are connected to hidden layer neurons over sensory gain factors. We willsee that the sensory gain factors in�uence the Kalman gain KKF.

4.4.2 Connection weights between HL neurons for dynamic systems

Let's assume we have a dynamic system with a system matrix A ∈ RS×S and a controlmatrix B ∈ RS×C . The question is: How can we �nd a dynamic system function g(x)with which we can calculate the weight matrix Whl as discussed in subsection 4.2.5?First we divide the value x � which a hidden layer activity ahl encodes � into componentsxs ∈ RS×1 and components xc ∈ RC×1:

x =

(xs

xc

)∈ RD×1 (4.51)

xs is the system state which is encoded in the system state subspace of the hidden layerand xc is the control input which is encoded in the control input subspace of the hiddenlayer. So the system state space is represented by the �rst S dimensions of the hiddenlayer and the control input state space is represented by the remaining C dimensions.The dynamic system can be written as follows

xs = A xs + B xc (4.52)

and we get an approximation for the continuous di�erential equation using the explicitEuler method:

xk+1s = xks + h ·

(A xks + B xkc

)(4.53)

xk+1s = (hA + I) xks + hB xkc (4.54)

If not otherwise mentioned we set the step width h to 1. I introduce a combined systemmatrix L such that the dynamic system function g(xk) is de�ned as follows:

g(xk) = L · xk (4.55)

We have to consider that g(xk) should only set the new system state xk+1s while com-

ponents of the control input subspace xk+1c should be set to zero:

xk+1 = L · xk (4.56)(xk+1

s

0

)= L ·

(xksxkc

)(4.57)

Under these considerations we can construct the system matrix L as follows:

L =

((A + I) B

OA OB

)∈ RD×D (4.58)


I ∈ RS×S (4.59)

OA ∈ RC×S (4.60)

OB ∈ RC×C (4.61)

OA and OB are zero matrices whose entries are all set to zero and I is an identity matrix.Now we have the dynamic system function g(x) and we can calculate the weight matrixWhl. However we cannot do this exactly like in subsection 4.2.5. We have to considerthat the new value xk+1 � which is encoded by the hidden layer activity ak+1

hl � muststay invariant in the control input subspace. Otherwise we won't be able to encode newcontrol values xk+1

c into ak+1hl . Consequently we don't want that ak+1

hl encodes

xk+1 6=(

xk+1s

0

)(4.62)

but we want that ak+1hl encodes

xk+1 =

(xk+1

s

∗

)(4.63)

which is invariant in the control input space. ∗ denotes an unde�ned entry. This leads toa ak+1

hl which isn't bell shaped in the D dimensional hidden layer space, but which is bellshaped in the S dimensional system state subspace. In the D dimensional hidden layerspace ak+1

hl has the shape of a ridge. To a achieve such a behavior where the propagatedhidden layer activity remains invariant in the control input space we calculate the weightmatrix Whl slightly di�erent to eq. 4.31. I introduce a selection matrix Ss which setscomponents which doesn't belong to the system state subspace to zero:

Ss =

(I O1

OT1 O2

)∈ RD×D (4.64)

I ∈ RS×S (4.65)

O1 ∈ RS×C (4.66)

O2 ∈ RC×C (4.67)

O1 and O2 are zero matrices whose entries are all set to zero and I is an identity matrix.Now we �nally can calculate Whl:

hlwkl = fact,hl

(L · lphl − Ss · kphl

)(4.68)

hlwkl = KL · exp

(KW

[D∑i=1

cos([

L · lphl − Ss · kphl

]i

)−D

])(4.69)

4.4.3 Propagate the dynamic system in the hidden layer

In the previous subsection 4.4.2 we have seen that for dynamic systems the hiddenlayer space is divided into a system state subspace (the �rst S dimensions) and into a


control input subspace (the last C dimensions). The lateral connection weights of thehidden layer are chosen is such a way that after each iteration step the propagatedhidden layer activity becomes invariant (unde�ned) in the control input subspace. Thefollowing question now arises: How can we encode a new control value xkc into a such acontrol invariant hidden layer activity akhl?First we calculate the combined hidden layer activity ahl, ctrl which encodes the controlinput xkc and which is invariant in the system state subspace. We start with a set ofpopulation code activities capc, ctrl which encodes xc and we calculate for each populationcode activity capc, ctrl an individual hidden layer activity cahl, ctrl:

[cahl, ctrl]i = [capc, ctrl]ihl(i, S+c) ∀i ∈ 1 . . . P ∧ ∀c ∈ 1 . . . C (4.70)

We notice that each individual hidden layer activity cahl, ctrl represents a multidimen-sional ridge in the hidden layer space and is invariant in all the other D− 1 dimensions.Now we combine the individual hidden layer activities cahl, ctrl to a combined controlinput activity ahl, ctrl by multiplying the activities cahl, ctrl element-wise [13]:

[ahl, ctrl]i =

C∏c=1

[cahl, ctrl]i (4.71)

ahl, ctrl =C©c=1

cahl,ctrl = 1ahl, ctrl ◦ · · · ◦ Cahl, ctrl (4.72)

This formalism we can abbreviate by using the Hadamard product operator ◦ which de-�nes that elements of a vector or matrix are multiplied element-wise with correspondingelements of the other vector or matrix. The resulting combined control input activityahl, ctrl is bell-shaped with respect to the control input subspace and is invariant withrespect to the system state subspace of the hidden layer.Now that we have the combined control input activity akhl, ctrl we can use it to encode

the control input value xkc into the control invariant hidden layer activity akhl and weget akhl, in [13]:

akhl, in = akhl ◦ akhl, ctrl (4.73)

akhl, in now encodes the system state and the control input for the iteration step k:

akhl, in ←→ xk =

(xksxkc

)(4.74)

Putting everything together we can propagate a dynamic system in the following way:

ak+1hl = h

(akhl ◦ akhl, ctrl

)(4.75)

4.4.4 Propagate the Kalman �lter in the hidden layer

We will see in section 5.2 that a Kalman �lter estimates system states in a recursiveway. Therefore it uses to two steps � the prediction step and the update step.


The prediction step uses a dynamic system to calculate a prediction −xks for a systemstate xks :

−xks = Φk +xk−1s + Bk xkc (4.76)

The dynamic system of the prediction step has a combined system matrix L so that(−xks0

)= L ·

(+xk−1

s

xkc

). (4.77)

The second step of the Kalman �lter � the update step � uses a system state feedbackxkfb to update the predicted system state −xks so that we get the �nal estimated systemstate +xks :

+xks = −xks + KKF(t) ·(xkfb − −xks

)(4.78)

+xks = (I−KKF(t)) · −xks + KKF(t) · xkfb (4.79)

We observe that the Kalman gain matrix KKF(t) decides which contribution gets moreweight � the prediction −xks or the feedback xkfb. The question now is: How can we realizesuch a system in a population code based neural network?The prediction step only consists of a dynamic system and can be implemented inthe same way as discussed in subsection 4.4.3. We just need to determine the combinedsystem matrix L such that it implements the desired prediction model. Then we calculatea hidden layer weight matrix Whl using eq. 4.69 and use the propagation function h(akhl)to calculate the activity −akhl which encodes −xks [13]:

−akhl = h(

+ak−1hl ◦ akhl, ctrl

)(4.80)

+ak−1hl encodes +xk−1

s of the previous iteration step and is invariant in the control inputspace:

−akhl ←→ −xk =

(−xks∗

)(4.81)

+ak−1hl ←→ +xk−1 =

(+xk−1

s

∗

)(4.82)

akhl, ctrl ←→ xkctrl =

(∗xkc

)(4.83)

Let's have a closer look at how we can realize the update step. First we convert the Spopulation code activities sapc, fb which encode the feedback value xfb into individualhidden layer activities sahl, fb:

[sahl, fb]i = [sapc, fb]ihl(i, s) ∀i ∈ 1 . . . P ∧ ∀s ∈ 1 . . . S (4.84)


The activities jahl, fb are then combined to a combined feedback activity ahl, fb [13]:

[ahl, fb]i =

S∑j=1

λj(t)[jahl, fb

]i

∀i ∈ 1 . . . P (4.85)

ahl, fb =S∑j=1

λj(t)jahl, fb (4.86)

We notice that the individual hidden layer activities jahl, fb are summed up in a weightedmanner and each activity jahl, fb has its own sensory gain value λj(t). The sensory gainvalues λj(t) along with the maximum activity heights of jahl, fb and −akhl in�uence therealized Kalman gain matrix KKF(t). However [13] doesn't introduce a principle or rulehow sensory gain values λj(t) de�ne KKF(t) so we don't know how to choose the λj(t)to get a desired KKF(t). Nevertheless I will deduce a principle for a Kalman �lter with1D system states in chapter 6.To realize the update step of the Kalman �lter in a hidden layer we just have to addthe combined feedback activity ahl, fb to the predicted activity −akhl [13]

+akhl = −akhl + akhl, fb . (4.87)

We summarize that the propagation of a Kalman �lter in the hidden layer can bedescribed as:

akhl = h

(ak−1

hl ◦C©c=1

cakhl, ctrl

)+

S∑j=1

λj(t)jakhl, fb (4.88)

akhl = h(ak−1

hl ◦ akhl, ctrl

)+ akhl, fb (4.89)

5 State-of-the-art attitude estimators

We have seen in section 3 that we have to combine sensory data to get reliable estima-tes for RPY angles. Each sensor alone cannot deliver reliable estimates. The process ofcombining sensory data to a more reliable and accurate result is called sensor fusion.Extensive research in this area has resulted in many algorithms and we can get an over-view of the most commonly used algorithms in [14]�[16].Most sensor fusion algorithm base upon three very basic principles � on a principle cal-led complementary �ltering, on a principle called Bayesian inference and on a principlecalled observer theory.In the following I want to give you an overview of state-of-the-art sensor fusion algo-rithms and explain brie�y the fusion principles they use. After the overview we will havea closer look at Kalman �lters and how we can use them to fuse sensory informationof the quadrotor to a reliable estimate for roll and pitch angles. In this work I focuson Kalman �lters because I plan to implement one into a population code network andshow that such an implementation works with real sensor data.

5.1 Overview

Complementary �ltering works with sensory signals with contain the same informa-tion but have complementary spectral characteristics. For example an accelerometersignal is very accurate in the low frequency spectrum but contains a lot of noise in thehigh frequency spectrum while on the other hand a gyroscope signal is very accuratein the high frequency spectrum but is unreliable in the low frequency spectrum. Bothsensor signals contain the same information as the signals can be used to calculate theattitude � the roll and pitch angles of an quadrotor. The most commonly used fusionalgorithms which take advantage of complementary �ltering are the linear complemen-tary �lter (LCF) [15] and the nonlinear complementary �lter (NCF) [15], [17], [18].Both �lters are used in quadrotor systems to estimate the attitude of the quadrotor.Practical applications show that complimentary �lters work and deliver reasonable butnot necessarily optimal estimation results. Often it's not easy to �nd usable parametersfor complimentary �lters as these parameters often depend on sensor properties whichchange with environment and time.

The linear complimentary �lter is a one-dimensional �lter and can separately fusesignals of the accelerometer and the gyroscope for roll and pitch angles [15]. A parameterof the LCF de�nes the so called crossover frequency. At this frequency the dominance ofthe two sensors in the frequency spectrum changes. For example for frequencies above thecrossover frequency the gyroscope should dominate as it is more reliable in this frequency

56 5 State-of-the-art attitude estimators

spectrum. The linear complementary �lter can use an integrator to compensate angulardrifts of the gyroscope.

The nonlinear complimentary �lter is a an extension of the LCF to the non-linearspecial orthogonal group SO(n) [15], [17], [18]. Instead of estimating the roll, pitch andyaw angles separately this �lter estimates the whole rotation matrix at once. This rotati-on matrix de�nes the rotation of the object frame (of the quadrotor) with respect to theworld frame. To estimate the rotation matrix the NCF introduces a di�erential matrixequation. The NCF algorithm can make use of sensory data of the accelerometer, thegyroscope and the magnetometer and several parameters de�ne crossover frequenciesfor the sensors the algorithm uses. Although the NCF algorithm has only recently beenintroduced as attitude estimator it is already successfully used in quadrotor applicationswhere it works accurately and reliably [15]. Because of the underlying very complex mo-del and its high dimensionality the NCF is computationally expensive. The parametershave to be tuned by hand and �nding good parameters often proves di�cult.

Bayesian inference is a stochastic theory which de�nes how information can be fusedin a stochastic optimal way [14], [16], [19]. Bayesian inference is only a theory and cannotdirectly be used for practical implementations. However many sensor fusion algorithmbases on this theory and make use of it. For example the Kalman �lter (KF), theextended Kalman �lter (EKF) and the particle �lter base on Bayesian inference andcombine it with and observer theory and internal models.

The observer theory describes the principle how system states of a real dynamicsystem can be estimated by using a model of the real system in combination with asystem state feedback [14]. The observer theory is used by Kalman �lters and extendedKalman �lters.

The Kalman �lter [14], [16] uses a linear system model and observer theory to estima-te system states in a stochastic optimal way . The Kalman �lter assumes independentGaussian noise sources and estimates system states in a recursive way. Each iterationstep consists of two steps � the prediction step and the update step. The prediction stepuses a linear system model (process model) to predict the system state of the real systemfor the current iteration step. The update step uses a measurement model to calculatea system state feedback which is then used to update the predicted system state to its�nal stochastic optimal estimate. The Kalman �lter uses several parameters (like cova-riances and Kalman gains) which are calculated online. For many implementations theseparameters tend to be constant so in order to save computing power these parameterscan be set to constant values without risk to deteriorate the performance of the Kalman�lter. However when the parameters are calculated online the Kalman �lter often evenworks without a proper initialization. Big di�erences between the predicted system stateand the true system state over short time periods don't in�uence the performance ofthe Kalman �lter in a negative way. So overall the Kalman �lter is a very robust �lterbut can implement only linear models.

5.2 The generic Kalman Filter (KF) 57

Many applications show that a Kalman �lter can successfully estimate quadrotor atti-tudes [20]. After the overview we will have a close look at how a Kalman �lter can beused to estimate quadrotor attitudes.

The extended Kalman �lter [15], [16] extends the Kalman �lter to nonlinear systemmodels (process models) and nonlinear measurement models . The EKF uses the sameprinciples as the Kalman �lter and approximates the nonlinear models by linearizingthem along the predicted trajectory of system states. For the EKF it is essential that thepredicted trajectory of system states is close to the true trajectory because even smallerrors can result in big approximation errors. If the assumed system state trajectoryis too far away from the true trajectory then the EKF has a very poor estimationperformance or even becomes unstable. For this reason the EKF must be initializedproperly and small inaccuracies in model or noise parameters can lead to a very poorperformance of the EKF. As the parameters of the EKF don't tend to be constant theymust be calculated online and thus the EKF needs signi�cantly more computations thanthe normal Kalman �lter. As we have seen in chapter 3 the EKF attitude estimatorimplementation of the PX4 quadrotor shows a very good estimation performance.

Particle �lters [14], [19] implement the Bayesian inference principle in a most genericway as particle �lters only base on an internal model and probability distributions. Par-ticle �lters can handle nonlinear models and non Gaussian noise distributions. Particle�lters are more distributed than Kalman �lters and don't explicitly rely on feedback [14].Overall particle �lters can show the same integration performance as Kalman �lters buthave an extremely high computational cost.

5.2 The generic Kalman Filter (KF)

Let's brie�y summarize how a Kalman �lter for an arbitrary linear model looks like[16],[20]. Note that in the following all predictions are marked with an upper-left minus signand all updates are marked with an upper-left plus sign. Each Kalman �lter uses twolinear models � the process model (system model) and the measurement model (sensormodel). The process model is a model of a real world system and we use this model topredict a new system state x:

xk = Φk xk−1 + Bk uk + wk (5.1)

Φ is the state transition matrix, B is the control input matrix, u is the control inputvector and w is the white Gaussian process noise of the process model and the controlinput.The measurement model describes how the sensor measures the system state of the realworld system:

zk = Hk xk + vk (5.2)


z is the feedback input which was measured by a sensor, H is the observation matrixwhich maps the system state space to feedback measurements and v is the white Gaus-sian measurement noise of the measurement model and the feedback input.A Kalman �lter estimates the system state x in a recursive manner and each iterationstep consists of a prediction step and an update step.

The prediction step First we predict the system state x and get a new prediction −xk.

−xk = Φk +xk−1 + Bk uk (5.3)

We notice that we need the previous estimation of the system state +xk−1 to calculatea new prediction. If we want to calculate the Kalman gain Kk

KF online instead of usinga constant one then we also have to predict the error covariance matrix −Pk:

−Pk = Φk +Pk−1 (Φk)T + Qk (5.4)

+Pk−1 is the updated error covariance matrix of the previous step. The predicted er-ror covariance matrix −Pk describes the predicted covariance of the error between theprediction of the system state −xk and the true system state xk

−Pk = cov(xk − −xk) (5.5)

and Q is the covariance matrix of the process noise wk:

Qk = E[wk (wk)T ] . (5.6)

The Kalman gain For most Kalman �lter implementations elements of the Kalmangain matrix Kk

KF tend to stay constant [16]. However we can calculate KkKF online

using the predicted error covariance matrix −Pk, the observation matrix Hk and thecovariance matrix Mk of the measurement noise v:

KkKF = −Pk (Hk)T

(Hk −Pk (Hk)T + Mk

)−1(5.7)

Mk = E[vk (vk)T ] (5.8)

We notice that even if we calculate the Kalman gain matrix KKF online we have toknow at least the stochastic properties of the process noise and the measurement noise.

The update step As soon as we have all the predictions and the Kalman gain matrixwe can start with the update step. First we update the system state to its �nal estimate:

+xk = −xk + KkKF

(zk −Hk −xk

)(5.9)

+xk =(I−Kk

KFHk)

−xk + KkKF zk (5.10)

5.3 The z-axis estimator (ZAE) 59

We see right away that the Kalman gain matrix decides which contribution to theupdated system state gets more weight. For small entries in the Kalman gain matrix theupdated system state is almost equal to the predicted system state and the feedbackhas almost no contribution. Finally we calculate the updated error covariance matrixfor the error between the true system state xk and the updated system state +xk:

+Pk =(I−Kk

KF Hk)

−Pk (5.11)

+Pk = cov(xk − +xk) (5.12)

5.3 The z-axis estimator (ZAE)

The z-axis estimator (ZAE) is introduced by [20]. This estimator uses a Kalman �l-ter and fuses sensory information of the accelerometer and the gyroscope. The z-axisestimator reliably estimates the roll and pitch angles and the linear acceleration of qua-drotors.

The idea of the ZAE is that instead of estimating the roll and pitch separately, the ZAEestimates the z-axis of the initial object frame OF0 with respect to the current objectframe OFk at time step k. We name this z-axis kz

0 using the naming conventions weintroduced in subsection 3.1.3. We also know from subsection 3.1.4 that the normalizedrotated gravitational acceleration vector akrot � which is also in coordinates of objectframe OFk � is equal to kz

0. As we have learned in subsection 3.1.4 kz0 can be directly

used to calculate roll and pitch angles so when the ZAE delivers accurate and reliableestimates for kz

0 we also get reliable and accurate estimates for roll and pitch angles.From the equations 3.22 and 3.23 we remember the relationship between the z-axis kz

0

and the roll and pitch angles φk and θk:

φk = arctan2([kz

0]2,[kz

0]3

)(5.13)

θk = −arctan2([kz

0]1,[kz

0]3

)(5.14)

The process model describes how we can calculate a prediction −k z0 for the true z-axis

kz0 at time step k. To get a description for a process model as introduced in section 5.2

we have to �nd a transition matrix Φk such that we can write

−k z0 = Φk · +k−1z

0 . (5.15)

We notice that the z-axis kz0 is the system state which we want to estimate by applying

the generic Kalman �lter principle. We also see that we don't have a separate controlinput because the process model uses a transition matrix Φk which already includes thecontrol input.Let's try to establish the process model for the ZAE and �nd its transition matrix Φk.From subsection 3.1.2 we know that the object frame OF0 and the object frame OFk


are just rotated against each other and that for coordinate transformations between thetwo object frames we can use the rotation matrix R(αkRPY) = R(φk, θk, ψk):

kz0 = RT (αkRPY) · 0z0 (5.16)

We cannot use this relationship in a direct manner to determine Φk. For the predictionmodel we only have gyroscope measurements αkRPY as control inputs and we don't know

0z0 � the z-axis z0 in the initial object frame OF0 � but only an estimate of k−1z

0 � thez-axis z0 in object frame OFk−1. So how can we get −

k z0 for a given αkRPY and k−1z0?

From a di�erent point of view Φk is just a transformation matrix which transforms thez-axis k−1z

0 in object frame OFk−1 of the previous step to a z-axis −k z0 in the predicted

object frame OFk. So Φk is a kind of di�erential rotation matrix which transformscoordinates from object frame OFk−1 to object frame OFk for given measurement ofthe gyroscope αkRPY:

−k z0 = dRT (αkRPY) · +k−1z

0 (5.17)

Let's �nd a formula for dRT (αkRPY). A di�erential rotation matrix dR is de�ned asfollows:

dR = I + [ω]× dt (5.18)

[ω]× is a skew-symmetric matrix of the Lie algebra and is the derivative of a rotationmatrix or in other words the di�erence between two rotation matrices for an in�nite-simally small time interval dt. ω is the angular velocity at this time interval dt andthe operator [ · ]× de�nes a cross-product between to vectors described as matrix vectormultiplication:

v ×w = [v]× ·w (5.19)

[v]× =

0 − [v]3 [v]2[v]3 0 − [v]1− [v]2 [v]1 0

(5.20)

[v]× = −([v]×

)T(5.21)

With this information we know how Φk = dRT (αkRPY) looks like:

dRT (αkRPY) =

(I +

[αkRPY

]×

dt

)T= I−

[αkRPY

]×

dt (5.22)

As result we get for the transition matrix Φk

Φk =

(I−

[αkRPY

]×

dt

)(5.23)

and the process model is

−k z0 =

(I−

[αkRPY

]×

dt

)· +k−1z

0 . (5.24)


We know from section 5.2 that we need a process noise covariance matrix to calculate aprediction for the error covariance matrix −Pk. [20] suggests to use the following processnoise covariance matrix Qk:

Qk = −(dt)2[+k−1z

0]× ·Σgyro ·

[+k−1z

0]× (5.25)

Σgyro = Iσ2gyro (5.26)

The measurement model uses accelerometer measurements aknet. If we subtract thelinear acceleration ak−1

lin of the previous iteration from aknet then we get a predictedrotated gravitational acceleration −akrot:

−akrot = aknet − cA · ak−1lin (5.27)

We can use −akrot as measurement value for the measurement model and thus get afeedback for kz

0:

−akrot = H kz0 (5.28)

H = I g (5.29)

For the online calculation of the Kalman gain matrix KkKF we need to know the measure-

ment covariance matrix Mk (see section section 5.2). For calculating Mk, [20] introducesthe following scheme:

Mk = Σacc, net + Σacc, lin (5.30)

Σacc, net = Iσacc, net (5.31)

Σacc, lin =1

3c2

A

∥∥∥ak−1lin

∥∥∥2· I (5.32)

The recursive estimator algorithm of the ZAE implements the Kalman �lter andestimates z-axes kz

0. It uses the introduced process and measurement models in com-bination with generic principles of the Kalman �lter and we get the following ZAEalgorithm:

1. Calculate the transition matrix Φk and predict the system state −k z0:

Φk =

(I−

[ωk]×

dt

)(5.33)

−k z0 = Φk · +k−1z

0 (5.34)

2. Calculate the process noise covariance matrix Qk and predict the error covariancematrix −Pk:

Qk = −(dt)2[+k−1z

0]× · Iσgyro ·

[+k−1z

0]× (5.35)


−Pk = Φk +Pk−1 (Φk)T + Qk (5.36)

3. Predict the rotated gravitational acceleration −akrot for an acceleration measure-ment aknet:


4. Determine the measurement noise covariance matrix and calculate the Kalmangain matrix Kk

KF:

Hk = I g (5.38)

Mk = Iσacc, net +1

3c2

A

∥∥∥ak−1lin

∥∥∥2· I (5.39)

KkKF = −Pk (Hk)T

(Hk −Pk (Hk)T + Mk

)−1(5.40)

5. Update the error covariance matrix +Pk :

+Pk =(I−Kk

KF Hk)

−Pk (5.41)

6. Update and normalize the system state +k z0:

+k z0 = −

k z0 + KkKF ·

(−akrot −Hk −

k z0)

(5.42)

+k z0 =

1∥∥+k z0

∥∥ · +k z0 (5.43)

7. Calculate the linear acceleration aklin:

aklin = aknet − g +k z0 (5.44)

8. Calculate the roll and pitch angles φk and θk:

φk = arctan2([

+k z0

]2,[+k z0

]3

)(5.45)

θk = −arctan2([

+k z0

]1,[+k z0

]3

)(5.46)

9. For the next iteration step we restart at step 1.

5.4 The simpli�ed angle estimator (SAE) 63

5.4 The simpli�ed angle estimator (SAE)

The simpli�ed angle estimator (SAE) is my own development and bases on the previous-ly discussed z-axis estimator (ZAE) (see section 5.3). Instead of estimating the z-axiswe now estimate the roll and pitch angles directly. As a result we get a simpler attitudeestimator which uses only two isolated 1D Kalman �lters instead of one more compli-cated 3D Kalman �lter. So I divide the centralized and complicated big problem solver� the 3D Kalman �lter � to several distributed and less complicated smaller problemsolving algorithms which consists mainly of two simple 1D Kalman �lters.The motivation for the development of the SAE is to �nd a low dimensional, distributedand simple attitude estimator which can easily be used as implementation model for apopulation code based neural network (PCNN). Indeed the SAE becomes very handyin chapter 6 when I need a simple model to show that a PCNN can be used to fusereal sensory signals to reliable roll and pitch estimates. In the following I assume forsimplicity that all accelerations are normalized to 1 g.The recursive estimation algorithm of the SAE uses three calculation blocks � the pre-processing block, the Kalman �lter block and the linear acceleration estimation block(see �gure 5.1).

preprocessingblock

Kalman filterblock

linear accelerationestimation block

accelerometer

gyroscope

aknet

dαkRP

dt

αkRP, fb

+αkRP

aklin

ak−1lin

+αkRP

Simplified angleestimator

Abbildung 5.1: Overview of the simpli�ed angle estimator with its three calculationblocks


The precalculation block encloses the calculations we need to calculate feedbackangles φkfb and θkfb for the two 1D Kalman �lters of the Kalman �lter calculation block.These feedback angles αkRP, fb base on the net linear acceleration aknet which is measured

by the accelerometer. As we know the net linear acceleration aknet contains the rotatedgravitational acceleration −akrot which we can use to determine αkRP, fb. To extract

−akrot

from aknet we subtract the linear acceleration of the previous iteration ak−1lin and get:


cA is tuning parameter. Then we can use −akrot and calculate the feedback angles αkRP, fb

φkfb = arctan2([

−akrot

]2,[−akrot

]3

)(5.48)

θkfb = −arctan2([

−akrot

]1,[−akrot

]3

). (5.49)

The Kalman �lter block uses two 1D Kalman �lters which fuses information of thegyroscope with information of the accelerometer.The Kalman �lters use angular velocities φk and θk of the gyroscope and previouslyestimated angles +φk−1 and +θk−1 which base on accelerometer measurements to predictnew angles for the current iteration step k:

−φk = +φk−1 + φk dt (5.50)

−θk = +θk−1 + θk dt (5.51)

The Kalman �lters update these predicted angles using Kalman gains KkKF, φ and K

kKF, θ

and feedback angles αkRP, fb:

+φk = −φk +KkKF, φ ·

(φkfb − −φk

)(5.52)

+θk = −θk +KkKF, θ ·

(θkfb − −θk

)(5.53)

As a result we get new estimates of the roll and pitch angles +αkRP for the currentiteration step k. Note that Kk

KF, φ and KkKF, θ can be calculated online or both Kalman

gains can be tuned to arbitrary constants such that the model shows good performance.We will see later in 7.1.1 that we can set Kalman gains to constant values because onlinecalculated Kalman gains also settle to constant values.

The linear acceleration estimation block uses current estimates of the roll and pitchangles +αkRP to update the rotated gravitational acceleration +akrot

+akrot = RT (+φk, +θk, 0) ·

001

(5.54)


+akrot =

− sin(+θk)sin(+φk) · cos(+θk)cos(+φk) · cos(+θk)

(5.55)

and uses +akrot in combination with the accelerometer measurement aknet to estimate thecurrent linear acceleration aklin:

aklin = aknet − cS · +arot (5.56)

cS is a tuning parameter.

The recursive estimator algorithm of the SAE algorithm can be described as follows:

1. Predict the rotated gravitational acceleration −akrot:


2. Calculate the feedback angles αkRP, fb:

φkfb = arctan2([

−akrot

]2,[−akrot

]3

)(5.58)


−akrot

]1,[−akrot

]3

)(5.59)

3. Predict and update the roll angle in the 1D Kalman �lter KFφ:

−φk = +φk−1 + φk dt (5.60)


(φkfb − −φk

)(5.61)

4. Predict and update the pitch angle in the 1D Kalman �lter KFθ:

−θk = +θk−1 + θk dt (5.62)


(θkfb − −θk

)(5.63)

5. Update the rotated gravitational acceleration +akrot:

+akrot =


(5.64)

6. Estimate the linear acceleration aklin:


7. For the next iteration we restart with step 1.

6 Attitude estimators in PC basedneural networks (PCNN)

6.1 Introduction

6.1.1 Deciding which attitude estimator to implement

In the previous chapters we have collected the knowledge we now need for the implemen-tation of a sensor fusion algorithm in a population code based neural network (PCNN).In chapter 4 we have discussed in a generic way of how we can implement mathemati-cal functions and Kalman �lters into PCNNs. In chapter 5 we have got an overview ofdi�erent state-of-the-art fusion algorithms and we have learned in detail how Kalman�lters can be used for attitude estimator algorithms.What I want to �nd out now is whether a collection of PCNNs can be used to imple-ment a sensor fusion algorithm which estimates roll and pitch angles of a quadrotorin real-time using raw and un�ltered real world sensor data of an accelerometer and agyroscope sensor. The goal is to verify and prove whether the PCNN can work in realworld sensor fusion applications.It is true that PCNN have already been veri�ed and used with arti�cial data but to myknowledge PCNNs have not been used with real world data to estimate the attitude ofan quadrotor. Up this point of my work it is not clear whether a sensor fusion algorithmimplemented in a PCNN will work with real sensor data.Reasons why a PCNN implementation might not work with real sensor data could bemeasurement noise, insu�cient accuracy of PCNNs, imperfect model implementationsin PCNNs, unstable error accumulations and many more.So as I don't now whether the proposed approach will work I decide to implement a sim-ple fusion algorithm in PCNNs. This simpli�es error analysis if something goes wrong.However if the approach works we are not limited to simple models and we can tryto implement more sophisticated sensor fusion algorithms which might have a betterperformance than the simple algorithm.For the implementation of a sensor fusion algorithm in PCNNs we have to consider thedimensionality of the algorithm. We cannot just use the next best sensor fusion algo-rithm and implement it into PCNNs � we have to analyze the algorithm and prepare itfor the implementation in PCNN. The principle of divide and conquer can work wondersand distributed problems greatly reduce the dimensionality problem of PCNNs.In chapter 5 I introduced the simpli�ed angle estimator (SAE) which I developed forthe use with PCNNs. I divided the high dimensional and concentrated z-axis estimator(ZAE) into smaller, distributed calculation blocks with a lower dimensionality.The rest of this chapter will explain how we can implement the SAE into PCNNs. From

68 6 Attitude estimators in PC based neural networks (PCNN)

now on we call the implementation of the SAE in PCNNs simply SAE-PC. In chapter7 I will feed the SAE-PC with real sensor data, validate if the SAE-PC works and isusable and �nally I will evaluate and compare the SAE-PC to the normal SAE andother state-of-the-art attitude estimators.

6.1.2 Discussing the approach

preprocessingblock

Kalman filterblock

linear accelerationestimation block

accelerometer

gyroscope

aknet

dαkRP

dt

αkRP, fb

+αkRP

aklin

ak−1lin

+αkRP

Simplified angleestimator

Abbildung 6.1: Overview of the simpli�ed angle estimator (SAE) which we want toimplement in PCNNs

We have decided in the previous subsection that I try to implement the SAE in PCNNs(SAE-PC). I introduced the SAE in subsection 5.4 and we can get an overview of therecursive processing loop in �gure 6.1.The SAE algorithm contains three calculation blocks � the preprocessing block, the Kal-man �lter block and the linear acceleration estimation block. Each calculation block is aset of mathematical formulas which we have introduced in subsection 5.4. A calculationblock can either be implemented in the traditional way by just using mathematical for-mulas or a calculation block can be implemented by using PCNNs. A calculation blockwhich just uses mathematical formulas is called reference block as it just calculates apart of the SAE. If a calculation block uses PCNNs then the PCNNs implement the

6.2 The preprocessing block 69

mathematical formulas of that calculation block and the block should calculate resultswhich are close to the result of the reference block.We see that the principle of calculation blocks allows us to implement the SAE into thePCNNs step by step. Whenever needed we can exchange a PCNN based block with areference block or vice versa. This makes life easier as problems can be located in a veryfast way.Now we will deduce PCNN implementations for the three calculation blocks step bystep and in the end we will have a PCNN implementation for the whole recursive loopof the SAE and thus the SAE-PC algorithm.

6.2 The preprocessing block

6.2.1 Topography of the neuronal network

Let's try to establish a macroscopic overview of the preprocessing block before we lookat the details of the PCNN implementation. We look now more at how many neurallayers we need and how these neural layers are connected to each other.From subsection 5.4 we know that the preprocessing block uses the following formulas:


φkfb = arctan2([

−akrot

]2,[−akrot

]3

)(6.2)


−akrot

]1,[−akrot

]3

)(6.3)

We structure these formulas into calculation layers CLi. Each calculation layer can berealized in a 3-layer PCNN. As we can see in �gure 6.2 the preprocessing block has twocalculation layers CLi. CL1 implements eq. 6.1 and CL2 implements eq. 6.2 and 6.3.From section 4.3 we remember that each 3-layer PCNN has an input layer block withinput layers ILi, an hidden layer block with hidden layers HLi and an output block withoutput layers OLi. Each output layer OLi has again several sub-layers SLi of which eachimplements a function for one output component of the output layer.


PC1 : 1 × 1 7→ N × 1PC2 : 1 × 1 7→ N × 1PC3 : 1 × 1 7→ N × 1

PC1 : 1 × 1 7→ N × 1PC2 : 1 × 1 7→ N × 1PC3 : 1 × 1 7→ N × 1

HL1 : N × 2 7→ P × 1HL2 : N × 2 7→ P × 1HL3 : N × 2 7→ P × 1

SL1 : P × 1 7→ N × 1SL2 : P × 1 7→ N × 1SL3 : P × 1 7→ N × 1

dcdAct

HL1 : N × 2 7→ P × 1HL2 : N × 2 7→ P × 1

SL1 : P × 1 7→ N × 1SL2 : P × 1 7→ N × 1

dcdAct

IL1

IL2

HL

HL

OL1

OL2

aknet

3 × 1

ak−1lin

3 × 1

Apc, rot

N × 3

Apc, net

N × 3

Apc, lin

N × 3

Ahl, out

P × 3Apc, rot

N × 3

Ahl, out

P × 2Apc, ang

N × 2

−akrot

3 × 1

αkRP, fb

2 × 1

CL1 : −akrot = ak

net − cA · ak−1lin

CL2 : αRP, fb =(

φkfb

θkfb

)=( arctan2

([−akrot]

2 ,[−ak

rot]

3)

−arctan2([−ak

rot]

1 ,[−ak

rot]

3))

Abbildung 6.2: The preprocessing block has two calculation layers CL1 and CL2; eachCL is a 3-layer network with input layers ILi, hidden layers HLi, outputlayers OLi; the output layers contain sub-layers SLi

The �rst calculation layer CL1 has two 3D inputs � the net linear acceleration aknet

and the linear acceleration of the previous step ak−1lin (see �gure 6.2). In �gure 6.2 we

see that the calculation CL1 has one 3D output � the predicted rotated gravitationalacceleration −akrot. So CL1 as a black box is a function which maps a 6-dimensionalinput space to a 3-dimensional output space. If we would implement CL1 as describedin section 4.3 we would need one 6D input layer with 6 input population codes, a 6Dhidden layer and an output layer with 3 sub-layers. This would not be a very intelligentthing to do as we should at all costs avoid high dimensional hidden layers (see subsection4.2.5). For example if we use N = 51 neurons per population code then a 6D hiddenlayer would have about 18 billion neurons.At a second look we can use the principle of divide and conquer and separate CL1 into3 calculations � one for each vector dimension. We have then three 2D to 1D mappingswhich can easily implemented into a PCNN (see �gure 6.2). The PCNN has 2 inputlayers which map the two 3D input vectors to two population code activity matricesApc, net and Apc, lin. The hidden layer block consist of three 2D hidden layers HLi whichencode the corresponding vector elements [anet]i and [alin]i into a common hidden layerspace and thus into a single hidden layer activity pattern iahl. All the three hiddenlayer activity patterns iahl together are combined to a hidden layer activity matrix Ahl.The components [anet]i and [alin]i which are now encoded in a hidden layer activity iahl

are then put to sub-layers SLi. Each sub-layer SLi usesiahl and a weight matrix iWpc

to calculate [anet]i and [alin]i to the desired result [arot]i which is then encoded in thepopulation code of SLi. The population code activities iapc, rot of all sub-layers SLi areorganized in the population code activity matrix Apc, rot. Finally we can decode Apc, rot


to the output −arot.

The second calculation layer CL2 directly uses the population code activity Apc, rot

of calculation layer CL1 as input. Consequently CL2 doesn't need an input layer toconvert −arot into population code activities iapc, rot (see �gure 6.2). Furthermore eq.6.2 and 6.3 only have 2D inputs so that two 2D hidden layer are su�cient to encodethe inputs of these equations in a common space. HL1 encodes [−arot]2 and [−arot]3 in acommon hidden layer activity pattern 1ahl while HL2 encodes [−arot]1 and [−arot]3 in acommon hidden layer activity pattern 2ahl. Sub-layer SL1 realizes eq. 6.2 and uses 1ahl

and 1Wpc to calculate the population code activity apc, φ. Sub-layer SL2 respectivelyrealizes eq. 6.3 and uses 2ahl and

2Wpc to calculate the population code activity apc, θ.The population code activities iapc, ang are organized in the population code activitymatrix Apc, ang which is then decoded to the feedback angles αRP, fb.

6.2.2 Predict the rotated gravitational acceleration

IL1

IL2

HL1 : g(x) = x

HL2 : g(x) = x

HL3 : g(x) = x

SL1 : 1r(x) = [x]1 − cA · [x]2

SL2 : 2r(x) = [x]1 − cA · [x]2

SL3 : 3r(x) = [x]1 − cA · [x]2

dcdAct

x =([

aknet

]1

[ak−1

lin]

1)T

x =([

aknet

]2

[ak−1

lin]

2)T

x =([

aknet

]3

[ak−1

lin]

3)T

OL

N × 1

1apc, net

N × 1

2apc, net

N × 1

3apc, net

N × 1

1apc, lin

N × 1

2apc, lin

N × 1

3apc, lin

P × 1

1ahl

P × 1

2ahl

P × 1

3ahl

N × 1

1apc, rot

N × 1

2apc, rot

N × 1

3apc, rot

3 × 1ak

net

3 × 1ak−1

lin

3 × 1

−akrot

Abbildung 6.3: The calculation layer to predict the rotated gravitation acceleration−akrot using 3-layer neural network of population codes

Now that we have a macroscopic view and know how the di�erent neural layers in acomputation layer are connected to each other we can discuss how the di�erent neurallayers have to be designed.In �gure 6.3 we get a more detailed view of the calculation layer CL1 which calculates thepredicted rotated gravitational acceleration ak−1

rot . All the neurons of the input layer, thehidden layer and the output layer use the same type of preferred values. The populationcodes use the same amount of neurons N which are distributed equally over a commonrange. For example we can useN = 51 neurons per population code which have preferredvalues in the range of −2 to 2. Then every preferred value has a distance of 0.0800 tothe next preferred value. Each dimension of the multidimensional preferred values ofhidden layer neurons follows the same scheme as the preferred values of the population


codes. They are also equally space over a common range.After the input vectors aknet and ak−1

lin have been encoded into 6 separate populationcode activities these population code activities are fed into the three 2D hidden layers.All three hidden layers use the same lateral connection weight matrix Whl which justimplements a stabilizing dynamic system function g(x) = x. Note that the weight matrixWhl only has to be calculated once and this calculation can be done o�ine. The threehidden layer activities iahl encode di�erent combinations of the 6 input components.For example the stabilized hidden layer activity 1ahl of hidden layer HL1 encodes

x =([

aknet

]1

[ak−1

lin

]1

)Twhich is exactly the input value x that sub-layer SL1 needs to calculate

[−akrot

]1. We

see in �gure 6.3 that all the hidden layers HLi encode a di�erent set of the 6 inputcomponents but all the three sub-layers SLi implement the same function r(x). Thuswe can use the same input weight matrix Wpc for all the sub-layers SLi in the outputlayer. Wpc which realizes r(x) has to be calculated only once and the calculation canbe done o�ine.

The calculation algorithm for the PCNN implementation of calculation layer CL1 canbe summarized as follows:

1. De�ne the preferred value matrices Ppc for population codes of input layers ILiand sub-layers SLi using def. 4.3

2. De�ne the preferred value matrices Phl for hidden layers HLi using def. 4.22 andeq. 4.23

3. Calculate hidden layer matrix Whl which can be used for all hidden layers HLiusing eq. 4.31

4. Calculate a population code input weight matrix Wpc which can be used for allsub-layer SLi using eq. 4.45

5. Calculate population code activity matrices Apc, net and Apc, lin for inputs aknet

and ak−1lin using eq. 4.13

6. Feed the needed components of Apc, net and Apc, lin into hidden layers HLi usingeq. 4.26 and 4.39

7. Stabilize hidden layer activities iahl using eq. 4.35

8. Calculate population code activities iapc, rot of sub-layers SLi for stabilized hiddenlayer activities iahl using eq. 4.46

9. Decode Apc, rot to−akrot using eq. 4.16

10. For iteration k + 1 restart at step 5


6.2.3 Calculate the accelerometer based feedback angles

HL1 : g(x) = x

HL2 : g(x) = x

SL1 : 1r(x) = arctan2([x]1 , [x]2)

SL2 : 2r(x) = −arctan2([x]1 , [x]2)

dcdActx =

([−akrot

]2

[−akrot

]3)T

x =([−ak

rot]

1[−ak

rot]

3)T OL

N × 3Ak

pc, rot

N × 1

2akpc, rot

N × 1

3akpc, rot

N × 1

1akpc, rot

N × 1

3akpc, rot

P × 1

1ahl

P × 1

2ahl

N × 1apc, φ

N × 1apc, θ

1 × 1φk

fb

1 × 1θk

fb

Abbildung 6.4: The calculation layer to calculate the accelerometer based feedbackangles φkfb and θkfb

With the help of �gure 6.4 we get a more detailed view of the second calculation layerCL2 which calculates accelerometer based roll and pitch angles αRP, fb. These anglesare later used as feedback angles. Again we use the same properties for all the preferredvalues we use in this calculation layer. We don't need an input layer for the predictedrotated gravitational acceleration −akrot of CL1. We already get population code activitiesApc, rot for

−akrot from the output layer of CL1. As the two hidden layers only implementa stabilizing dynamic system g(x) = x in a 2D space we can use for both hidden layersthe same weight matrix Whl � the weight matrix we have already calculated for CL1.The two 2D hidden layers encode di�erent values x because the sub-layers need di�erentinput components for the calculation of φfb and θfb. We can also see in 6.4 that eachsub-layer SLi implements a di�erent function ir(x) so we need to calculate two di�erentinput weight matrices Wpc for the population codes of the the two sub-layers. Theweight matrix 1Wpc must realize

1r(x) and the weight matrix 2Wpc must realize2r(x).

As before the weight matrices iWpc need to be calculated only once and the calculationcan be done o�ine.


1. De�ne the preferred value matrices Phl for hidden layers HLi using def. 4.22 andeq. 4.23

2. Calculate the hidden layer matrix Whl which can be used for all hidden layersHLi using eq. 4.31

3. Calculate for each sub-layer SLi a population code input weight matrix iWpc usingeq. 4.45

4. Feed the needed components of Apc, rot into hidden layers HLi using eq. 4.26 and4.39


5. Stabilize hidden layer activities iahl using eq. 4.35

6. Calculate population code activities iapc, φ and iapc, θ of sub-layers SLi for stabi-lized hidden layer activities iahl using eq. 4.46

7. Decode apc, φ and apc, θ to αkRP, fb using eq. 4.16


6.3 The Kalman �lter block

6.3.1 Topography of the neural network

The Kalman �lter block uses two separate 1D Kalman �lters KFφ and KFθ. TheseKalman �lters use gyroscope data to predict roll and pitch angles and they use feedbackangles φfb and θfb of the previous block to update the prediction. From subsection 5.4we know that KFφ and KFθ use the following formulas:

−φk = +φk−1 + φk dt (6.4)


(φkfb − −φk

)(6.5)

−θk = +θk−1 + θk dt (6.6)


(θkfb − −θk

)(6.7)

To implement KFφ and KFθ into a PCNN we use a special hidden layer � which hasbeen introduced in section 4.4.4 � in combination with input layers which encode angularvelocities and feedback angles to population code activities apc, c and apc, fb (see �gure6.5). We can see in �gure 6.5 that we need such a special PCNN for each Kalman�lter KFφ and KFθ. The hidden layer HLi implements one whole Kalman �lter with itsprediction step (eq. 6.4 or 6.6) and its update step (eq. 6.5 or 6.7). The hidden layer spaceis a 2D space which encodes the currently estimated angle +φk or +θk and the controlinput φk+1 or θk+1 for the next iteration step. We know from section 4.4.4 that the lateralconnection weight matrix Whl of the hidden layer implements the dynamic system ofthe prediction step while a simple addition of hidden layer activities implements theupdate step. The Kalman gain depends on the heights of hidden layer activities � theheight of the feedback hidden layer activity ahl, fb and the height of the prediction hiddenlayer activity −ahl. The height of the prediction hidden layer activity −ahl is always onewhile the height of the feedback hidden layer activity ahl, fb is modulated using sensorygain factors. After the propagation of the hidden layer activity +ak−1

hl we get the newhidden layer activity +akhl which encodes a new estimation of the roll or pitch angle +φk

or +θk.

6.3 The Kalman �lter block 75

PC1 : 1 × 1 7→ N × 1

PC1 : 1 × 1 7→ N × 1

HL1 : N × 2 7→ P × 1 dcdActIL1

IL2

HL1

apc, c

N × 1

apc, fb

N × 1

+akhl

P × 1

PC1 : 1 × 1 7→ N × 1

PC1 : 1 × 1 7→ N × 1

HL1 : N × 2 7→ P × 1 dcdActIL1

IL2

HL1

apc, c

N × 1

apc, fb

N × 1

+akhl

P × 1

KFφ : −φk = +φk−1 + dφk

dtdt

+φk = −φk + KkKF, φ ·

(φk

fb − −φk)

KFθ : −θk = +θk−1 + dθk

dtdt

+θk = −θk + KkKF, θ ·

(θk

fb − −θk)

dφk

dt

1 × 1

φkfb

1 × 1

+φk

1 × 1

dθk

dt

1 × 1

θkfb

1 × 1

+θk

1 × 1

Abbildung 6.5: The Kalman �lter block consists of two Kalman �lters KFφ and KFθ;each Kalman �lter is implemented in a special hidden layer HLi; inputlayers ILi encode control and feedback input values to population codeactivities

6.3.2 Calculate recursive Kalman �lter estimations

IL1

λ

HL1 : g(x) =(

1 dt0 0

)· x dcdAct

x =(

+φk−1 dφk

dt

)T

dφk

dt

1 × 1

φkfb

1 × 1

apc, c

N × 1

apc, fb

N × 1λ · apc, fb

N × 1

+akhl

P × 1

+φk

1 × 1

Abbildung 6.6: The calculation layer for Kalman �lter KFφ of the Kalman �lter block

Now that we have gained an overview of the di�erent neural layers we need for thedesign of the Kalman �lter block we can have a closer look at the details (see �gure6.6). Note that I discuss only the details of Kalman �lter KFφ as both Kalman �ltersuse an identical processing structure (see �gure 6.5).


First the input values φk and φkfb need to be encoded into population code activitiesapc, c and apc, fb. The feedback activity apc, fb is scaled with a sensory gain factor λ. Wewill see in subsection 6.3.3 that the sensory gain factor λ sets the Kalman gain KKF ofthe PCNN. The scaled feedback activity λ · apc, fb and the control input activity apc, c

are then passed to the special 2D hidden layer HL1. The two dimensions of the hiddenlayer encodes the system state (+φk−1, −φk or +φk) and the control input φk. Lateralconnection weights Whl of HL1 implement a dynamic system g(x):

g(x) =

(1 dt0 0

)· x (6.8)

This dynamic system g(x) is equivalent to eq. 6.4 which describes the prediction step ofthe Kalman �lter KFφ. Before the prediction step can be executed in the hidden layerthe new control input φk needs to be encoded into the control input invariant hiddenlayer activity +ak−1

hl of the previous iteration step k − 1. The resulting hidden layer

activity akhl, in encodes now +φk−1 and φk. akhl, in can now be propagated through the

lateral connections of the hidden layer and we get a hidden layer activity −akhl whichencodes the predicted system state −φk and which is invariant in the control input space.Now we can add the scaled feedback activity in its hidden layer representation akhl, fb to

the current hidden layer activity −akhl and we get a new hidden layer activity +akhl.+akhl

is the updated hidden layer activity we get after the update step and +akhl encodes theupdated system state +φk. Note that +akhl remains invariant in the control input space.The calculation algorithm for the Kalman �lter in a PCNN is:

1. De�ne the preferred value matrix Ppc for population codes of the input layer IL1

using def. 4.3

2. De�ne the preferred value matrix Phl for hidden layer HL1 using def. 4.22 and eq.4.23

3. Calculate the hidden layer matrix Whl for hidden layer HL1 using eq. 4.69

4. Calculate population code activities akpc, c and akpc, fb for inputs φk and φkfb usingeq. 4.13

5. Feed akpc, c and akpc, fb into hidden layer HL1 using eq. 4.26 and get individual

hidden layer activities akhl, ctrl and akhl, fb

6. Scale akhl, fb with λ using eq. 4.86

7. Encode akhl, ctrl into the hidden layer activity +ak−1hl of iteration step k− 1 and get

akhl, in using eq. 4.73

8. Propagate hidden layer activity akhl, in using eq. 4.80 and get predicted system state

encoded in −akhl

9. Add feedback activity akhl, fb to hidden layer activity −akhl using eq. 4.87 and get

the updated system state encoded in +akhl

6.3 The Kalman �lter block 77

10. Decode +akhl and get +φk using eq. 4.16


6.3.3 From Kalman gain to network parameters

We have seen in subsection 4.4.4 that the update step in a PCNN is realized as a simplesum of hidden layer activities:

+akhl = −akhl + akhl, fb (6.9)

We also remember that this equation is equivalent to

+xks = −xks +KKF ·(xkfb − −xks

). (6.10)

For our Kalman �lters KFφ and KFθ the hidden layer encodes 2D values x. The systemstate xs is encoded in the �rst dimension and the control input xc is encoded in the seconddimension. After the prediction step we get the predicted system state −xs encoded in−akhl which is invariant in the control input space. −akhl is a ridge in the 2D space of thehidden layer with a maximum activity of 1:

KL, pred = scaleOfAct(−akhl, ahl, 0) = 1 (6.11)

akhl, fb is also invariant in the control input space and encodes the feedback value xkfbin the �rst dimension. akhl, fb is a ridge in the 2D hidden layer space and its maximumactivity is de�ned by the sensory gain factor λ:

KL, fb = scaleOfAct(akhl, fb, ahl, 0) = λ (6.12)

What we have to �nd out now is how we need to choose λ to realize a given Kalmangain KKF. Therefore I change λ and thus KL, fb while I set the encoded values −xks andxkfb to some arbitrary constant values. What I found out is that the receptive �eld RFand the activity scaling factors KL, pred and KL, fb in�uences the encoded result +xks . Athorough analysis shows that the summation of 2D hidden layer activities follows thefollowing rules:

+xks = −xks +KL, fb

KL, fb +KL, pred·(xkfb − −xks

)(6.13)

i�(xkfb − −xks

)≤ RF ∧ KL, fb

KL, fb +KL,pred≤ 0.5

+xks = −xks (6.14)


)� RF ∧ KL, fb

KL, fb +KL, pred≤ 0.5

+xks = xkfb +KL,pred

KL, fb +KL, pred·(−xks − xkfb

)(6.15)


i�(−xks − xkfb

)≤ RF ∧ KL,pred


+xks = xkfb (6.16)


)� RF ∧ KL, fb


For us eq. 6.13 is the most interesting rule for adding hidden layer activities −akhl andakhl, fb as it is identical to the update step of eq. 6.10. The Kalman gain KKF is then:

KKF =λ

λ+ 1i�

(xkfb − −xks

)≤ RF ∧ KKF ≤ 0.5 (6.17)

We will see in subsection 7.1.1 that he constraint KKF ≤ 0.5 is not an issue. UsuallyKKF is smaller than 0.5 by several orders of magnitude. In subsection 7.1.1 we will alsodiscuss what happens if xkfb − −xks is bigger than the receptive �eld RF . This happensquite often in real world applications. Because of noise we often have big di�erencesbetween prediction and feedback values.

6.4 The linear acceleration estimation block

6.4.1 Topography of the neural network

The linear acceleration estimation block implements two layers of calculations. Fromsubsection 5.4 we know:

+akrot =


(6.18)


Calculation layer CL1 implements eq. 6.18 and uses the newly estimated roll and pitchangles +φk and +θk to calculate an updated and more accurate version of the rotatedgravitational acceleration +akrot. Calculation layer CL2 implements eq. 6.19 and subtracts+akrot from the net linear acceleration aknet and gets aklin.

6.4 The linear acceleration estimation block 79

PC1 : 1 × 1 7→ N × 1PC2 : 1 × 1 7→ N × 1

HL1 : N × 2 7→ P × 1SL1 : P × 1 7→ N × 1SL2 : P × 1 7→ N × 1SL3 : P × 1 7→ N × 1

dcdAct

PC1 : 1 × 1 7→ N × 1PC2 : 1 × 1 7→ N × 1PC3 : 1 × 1 7→ N × 1

HL1 : N × 2 7→ P × 1HL2 : N × 2 7→ P × 1HL3 : N × 2 7→ P × 1

SL1 : P × 1 7→ N × 1SL2 : P × 1 7→ N × 1SL3 : P × 1 7→ N × 1

dcdAct

IL1HL1

OL1

IL2 HL2 OL2

αkRP

2 × 1

aknet

3 × 1

Akpc, rot

N × 3

+akrot

3 × 1

aklin

3 × 1

Apc, ang

N × 2ahl

P × 1Ak

pc, rot

N × 3

Apc, net

N × 3Ahl, out

P × 3Apc, lin

N × 3

CL1 : +akrot =

(− sin(+θk) sin(+φk) · cos(+θk) cos(+φk) · cos(+θk)

)T

CL2 : aklin = ak

net − cS · +arot

Abbildung 6.7: Calculation layer for the linear acceleration estimation block

The �rst calculation layer CL1 (see �gure 6.7) has a 2D input αkRP � the estimated rolland pitch angles from the Kalman �lter block � and a 3D output +akrot � the updatedrotated gravitational acceleration. The input layer IL1 has two population codes andencodes αkRP into an activity matrix Apc, ang. We feed the activity matrix Apc, ang intothe 2D hidden layer HL1 which then encodes the elements of αkRP in one common hiddenlayer activity ahl. The hidden layer activity ahl is used by the sub-layers SLi to calculateactivities iapc, rot for components

[+akrot

]iof the output +akrot. The sub-layers SLi uses

input weight matrices iWpc for their population codes and the weight matrices iWpc

are di�erent for each sub-layer SLi. The weight matrices iWpc are di�erent because eachcomponent

[+akrot

]iof the output needs to be calculated in a di�erent way (see eq. 6.18).

The population code activities of the output layer are organized in an activity matrixApc, rot which we can decode to +akrot.

The second calculation layer CL2 directly uses the activity matrix Apc, rot of calcula-tion layer CL1 (see �gure 6.7). Nevertheless CL2 needs for its calculations also net linearaccelerations aknet which the input layer IL2 encodes into the activity matrix Apc,net.We pass the activity matrices Apc, rot and Apc, net to a set of hidden layers HLi. Wenotice that the sub-layers of CL2 have only 2D inputs so a set of three 2D hidden layersHLi is su�cient. Each 2D hidden layer encodes input components

[aknet

]iand

[+akrot

]i

to a common hidden layer activity iahl and all the iahl together form the activity matrixAhl, out. The sub-layers of OL2 use Ahl, out in combination with a common input weightmatrix Wpc to calculate activities iapc, lin. All the activities iapc, lin together form theactivity matrix Apc, lin which we �nally can decode to the linear acceleration aklin.


6.4.2 Update the rotated gravitational acceleration

IL1 HL1 : g(x) = x

SL1 : 1r(x) = − sin([x]2)

SL2 : 2r(x) = sin([x]1) · cos([x]2)

SL3 : 3r(x) = cos([x]1) · cos([x]2)

dcdAct

x =(+φk +θk

)T

OL

+αkRP

2 × 1

1apc, ang

N × 1

2apc, ang

N × 1

ahl

P × 1

1apc, rot

N × 1

2apc, rot

N × 1

3apc, rot

N × 1

+akrot

3 × 1

Abbildung 6.8: The calculation layer to update the rotated gravitational acceleration+akrot in the linear acceleration estimation block

Let's have a closer look at calculation layer CL1 of �gure 6.7. which uses roll andpitch angles of the Kalman �lter block to calculate the updated rotated gravitationalacceleration +akrot (see �gure 6.8). First, population codes of the input layer IL1 encoderoll and pitch angles +φ and +θ of +αRP to population code activities 1apc, ang and2apc, ang. These population code activities are passed to the hidden layer HL1. Thehidden layer uses a lateral weight matrix Whl � which we can calculate o�ine � sothat HL1 implements a stabilizing dynamic system g(x) = x. The stabilized hidden

layer activity ahl encodes x =(

+φk +θk)T

and x is the input value for all the sub-layers SLi of the output layer OL1. So all three sub-layers can use the same hiddenlayer activity ahl. We observe in �gure 6.8 that each sub-layer implements a di�erentmathematical function thus we have to calculate for each sub-layer SLi a di�erent weightmatrix iWpc. Each weight matrix iWpc realizes a function ir(x). We need to calculatethe weight matrices iWpc only once and the calculation can be done o�ine.


1. De�ne the preferred value matrices Ppc for population codes of the input layerIL1 and the sub-layers SLi using def. 4.3

2. De�ne the preferred value matrix Phl for the hidden layer HL1 using def. 4.22 andeq. 4.23

3. Calculate the hidden layer weight matrix Whl for the hidden layer HL1 using eq.4.31

4. Calculate for each sub-layer SLi a population code input weight matrix Wpc usingeq. 4.45

5. Calculate population code activities 1apc, ang and 2apc, ang for +φk and +θk usingeq. 4.13

6.4 The linear acceleration estimation block 81

6. Feed 1apc, ang and 2apc, ang into hidden layer HL1 using eq. 4.26 and 4.39

7. Stabilize the hidden layer activity ahl using eq. 4.35

8. Calculate population code activities iapc, rot of sub-layers SLi for the stabilizedhidden layer activity ahl using eq. 4.46

9. Decode Apc, rot to+akrot using eq. 4.16


6.4.3 Calculate the linear acceleration

IL1 HL1 : g(x) = x

HL2 : g(x) = x

HL3 : g(x) = x

SL1 : 1r(x) = [x]1 − cS · [x]2

SL2 : 2r(x) = [x]1 − cS · [x]2

SL3 : 3r(x) = [x]1 − cS · [x]2

dcdAct

x =([

aknet

]1

[+akrot

]1)T

x =([

aknet

]2

[+akrot

]2)T

x =([

aknet

]3

[+akrot

]3)T

OL

aknet

3 × 1

Akpc, rot

N × 3

1apc, net

N × 1

2apc, net

N × 1

3apc, net

N × 1

1akpc, rot

N × 1

2akpc, rot

N × 1

3akpc, rot

N × 1

1ahl

P × 1

2ahl

P × 1

3ahl

P × 1

1apc, lin

N × 1

2apc, lin

N × 1

3apc, lin

N × 1

aklin

3 × 1

Abbildung 6.9: The calculation layer to calculate the linear acceleration aklin in the linearacceleration estimation block

Let's have a closer look at calculation layer CL2 (see �gure 6.9) where we use the result+akrot of calculation layer CL1 and the net linear acceleration aknet of the accelerometerto calculate the linear acceleration alin. We only need an input layer for aknet to getpopulation activities iapc, net because for +akrot we can use the activity matrix Arot ofcalculation layer CL1. We see in �gure 6.9 that we need three 2D hidden layers whichstabilize di�erent combinations of input population code activities to stabilized hiddenlayer activities iahl. All the hidden layers use the same stabilizing dynamic systemg(x) = x so we only need to calculate one lateral weight matrix Whl. A stabilized hidden

layer activity iahl encodes x =([

aknet

]i

[+akrot

]i

)T. The sub-layers of the output layer

all implement the same function r(x) so we need to calculate only one input weightmatrix Wpc.



1. De�ne the preferred value matrices Ppc for population codes of the input layerIL1 and the sub-layers SLi using def. 4.3

2. De�ne the preferred value matrix Phl for hidden layers HLi using def. 4.22 andeq. 4.23

3. Calculate the hidden layer weight matrix Whl which can be used for all hiddenlayers HLi using eq. 4.31

4. Calculate a population code input weight matrix Wpc which can be used for allsub-layers SLi using eq. 4.45

5. Calculate population code activities iapc,net for aknet using eq. 4.13

6. Feed iapc, net andiakpc, rot into hidden layers HLi using eq. 4.26 and 4.39

7. Stabilize the hidden layer activities iahl using eq. 4.35

8. Calculate population code activities iapc, lin of sub-layers SLi for stabilized hiddenlayer activities iahl using eq. 4.46

9. Decode Apc, lin to aklin using eq. 4.16


7 Evaluate and compare attitudeestimators

In this chapter I evaluate and compare di�erent attitude estimators which I introducedin the previous chapters. In subsection 3.2.5 I have already discussed and evaluated theperformance of the EKF attitude estimator of the quadrotor. So here I only use resultsof the quadrotor EKF for comparisions to the other attitude estimators.All the attitude estimators we analyze here use real and un�ltered sensory data of thesame quadrotor test �ight. The data is acquired using the t�og logging software (seechapter 4) and the attitude estimators are implemented in MATLAB using the self-written MATLAB libraries px4-lib and pc-lib. I use the px4-lib library (see chapter2) to import data of log �les generated by the t�og program and the pc-lib (see chapter4) to implement PCNNs of the SAE-PC algorithm (see chapter 6).

7.1 The z-axis estimator (ZAE)

The z-axis estimator (ZAE) bases on [20] and has been discussed in detail in subsection5.3. We remember that the ZAE uses a standard Kalman �lter to fuse sensory data ofthe accelerometer and the gyroscope to a new estimate +

k z0 � the z-axis of the initialobject frame OF0 with respect to the current object frame OFk. The ZAE calculates itsKalman gain matrix KKF(t) online and consequently uses Kalman gain factors whichchange over time and which insure a stochastic optimal fusion of sensory data. Let'sfeed the ZAE with real and un�ltered sensory data of a quadrotor test �ight and havelook at the elements of KKF(t). Afterwards we evaluate the performance of the ZAE.

7.1.1 Online Kalman gain calculation

The z-axis estimator (ZAE) calculates its Kalman gain matrix KKF(t) online and wehave learned in subsection 5.2 that the Kalman gain factors decide which model of theKalman �lter gets more weight and thus dominates the estimated result. Small entriesin KKF(t) mean that Kalman �lter results almost completely rely on the result of theprocess model � on the predicted system state � while the measurement model and thusthe system state feedback is almost neglected. Thinking about these two statements weexpect that the elements of KKF(t) stabilize to more or less constant values as soon asthe estimator has found the optimal compromise between prediction and feedback. In�gure 7.1 we see two entries KFk11(t) and KFk22(t) of KKF(t) which lie on the matrixdiagonal. All the other matrix entries are almost zero and don't change over time. Weobserve that KFk11(t) and KFk22(t) clearly converge to constant values and stay there.We observe also that at the beginning the predicted system state gets a lot of weight

84 7 Evaluate and compare attitude estimators

while after some time the feedback also gets some in�uence. Which model gets initiallymore weight is de�ned by the initialization of the Kalman �lter. The Kalman gain valuessettle to very small values of about 3.5× 10−4. This can be explained easily. From thedata analysis (see chapter 3) we know that sensory data of the accelerometer is extremelynoisy. The ZAE uses data of the accelerometer to calculate the system state feedback.We also know form data analysis that sensory data of the gyroscope is less noisy andthat the angular drift is only increasing very slowly. Considering these two aspects itmakes sense that the Kalman �lter calculates small Kalman gain values and thus reliesmore on gyroscopic data. The Kalman �lter needs the accelerometer only to compensatethe angular drifts of the gyroscope.

Kalm

angain

factor

t in sec.

Kalm

angain

factor

t in sec.

Kalman gain factor KFk22 :

Kalman gain factor KFk11 :

0 10 20 30 40 50 60

0 10 20 30 40 50 60

×10−4

×10−4

0

1

2

3

4

0

1

2

3

4

Abbildung 7.1: Entries on diagonal of Kalman gain matrix KKF(t) as calculated onlineby the z-axis estimator algorithm (ZAE); the diagonal entries settle toconstant values while all the other Kalman gain matrix entries are closeto zero over all times

7.1.2 Performance of the estimator

Let's evaluate the performance of the z-axis estimator (ZAE) by comparing its resultsto ground truth data (GT) and to results of the quadrotor's online attitude estimator(EKF-QR) (see subsection 3.2.5).If we compare �gures 7.2a and 7.2c to �gures 7.2b and 7.2d we notice that the EKF-QR is slightly better than the ZAE. However this is not that surprising as we haveto consider that the EKF-QR uses a more advanced nonlinear process model which


also makes use of magnetometer data. The EKF-QR runs on the quadrotor and thusgets raw data with a higher sample frequency in a more constant data rate with lessdata loss. We remember that we get test �ight data of the quadrotor via a wirelessconnection and the MAVLINK protocol doesn't guarantee that no data is lost. It mustalso be mentioned that high linear accelerations of the quadrotor (the quadrotor starts,changes and stops movements in an abrupt way) deteriorate the performance of theZAE. Whenever the quadrotor underlays high linear accelerations, feedback values ofthe accelerometer become unreliable and the estimator cannot compensate angular driftsand must rely on gyroscopic data only. However when high linear accelerations declinethe ZAE shows again a good performance.The fact that high linear accelerations deteriorate the performance of attitude estimatorsis not only true for the ZAE but also for all other estimators (like the SAE) which useaccelerometer data as system state feedback.

ZAE

GT

φis

rollangle

inrad

t in sec.


10 20 30 40 50 60 70

−0.2

−0.1

0

0.1

0.2

(a)

EKF-QR

GT

φis

rollangle

inrad

t in sec.


10 20 30 40 50 60 70

−0.2

−0.1

0

0.1

0.2

(b)


ZAE

GT

θis

pitch

angle

inrad

t in sec.


10 20 30 40 50 60 70

−0.2

−0.1

0

0.1

0.2

(c)

EKF-QR

GT

θis

pitch

angle

inrad

t in sec.


10 20 30 40 50 60 70

−0.2

−0.1

0

0.1

0.2

(d)

Abbildung 7.2: Evaluate performance of z-axis estimator (ZAE) and compare its resultsto results of the EKF which runs online on the quadrotor (EKF-QR); wededuced in subsection 3.2.5 that the ground truth system (GT) reportserroneous pitch angles between t = 18 s and t = 32 s

7.2 The simpli�ed angle estimator (SAE)

I introduced the simpli�ed angle estimator (SAE) in subsection 5.4. The SAE bases onthe z-axis estimator (ZAE) but directly estimates roll and pitch angles. Instead of usinga 3D Kalman �lter to estimate the z-axis, SAE uses two separate 1D Kalman �ltersKFφ and KFθ to estimate roll and pitch angles φk and θk. The SAE doesn't calculatethe Kalman gains online but uses constant Kalman gain values.In �gures 7.3a and 7.3b we can compare the SAE to ground truth data (GT) and observe


that the SAE delivers feasible results. If we compare the performance of the SAE to theperformace of the ZAE (see �gures 7.2a and 7.2c) then we notice that the performance ofthe two estimators is almost equal. This is considerable as one might await a signi�cantlyinferior performance of SAE as it uses constant Kalman gains, as it uses two separatedKalman �lters instead of one and as it completely neglects the yaw angles.

SAE

GT

φis

rollangle

inrad

t in sec.


10 20 30 40 50 60 70

−0.2

−0.1

0

0.1

0.2

(a)

SAE

GT

θis

pitch

angle

inrad

t in sec.


10 20 30 40 50 60 70

−0.2

−0.1

0

0.1

0.2

(b)

Abbildung 7.3: Evaluate performance of simple angle estimator (SAE) by comparing itsresults to ground truth data (GT); we deduced in subsection 3.2.5 thatthe ground truth system reports erroneous pitch angles between t = 18 sand t = 32 s


7.3 The simpli�ed angle estimator in PCNNs (SAE-PC)

In chapter 6 I described in a very detailed way how we can implement the simpli�edangle estimator (SAE) into a population code based neural network (PCNN). In thefollowing we will refer to this implementation by using the abbreviation SAE-PC.The most important thing I want to show with the SAE-PC is that such an implemen-tation in PCNNws can actually work with un�ltered data of real sensors on a real worldsystem. If this implementation works it is very probable that other, more sophisticatedfusion algorithm might work in PCNNs, too. As a PCNN implementation doesn't cal-culate with exact values but with activity patterns of population codes it is not surewhether such an implementation is stable at all. Before we evaluate the SAE-PC bycomparing its results to the results of other fusion algorithms, we have a look at howthe Kalman gain in a PCNNs is in�uenced by receptive �eld widths and di�erencesbetween predicted system states and feedback system states.

7.3.1 Kalman gain and receptive �elds

In subsection 6.3.3 we have seen that a Kalman �lter implementation in PCNNs doesn'texactly implement the update step of the Kalman �lter. Whenever the di�erence betweenthe predicted system state −x and the system state feedback xfb is bigger than thereceptive �eld RF then the Kalman gain KKF is decreasing until it becomes zero. Let'shave a closer look at this behavior of Kalman �lters in PCNNs.In �gure 7.4 we can observe how the Kalman gain KKF changes whenever the di�erencexdiff = xfb − −x becomes greater then the receptive �eld RF and in �gure 7.4a we seehow the updated system state +x should raise linearly for increasing di�erences xdiff .But for the PCNN implementation of a Kalman �lter KFpc

+x raises linearly withincreasing xdiff only at the beginning. When the di�erence gets close to the receptive�eld RF then +x starts to fall back to the predicted system state −x. Consequently KFpc

behaves like a normal Kalman �lter as long as xdiff is smaller than the receptive �eld.For di�erences xdiff bigger than RF the Kalman gain drops (see �gure 7.4b) until theupdate step of KFpc neglects feedback values completely. Then the updated system state+x becomes equal to the predicted system state −x. Thinking more about this behaviorwe realize that this behavior is desirable for the SAE. System state feedback gained byaccelerometer data is extremely noisy (see �gure 3.5) and a drop of the Kalman gainfor big di�erences between system state prediction and system state feedback improvesthe reliability of the SAE-PC. For smaller Kalman gains the system state feedbackgets smaller weight and the SAE-PC relies more on the gyroscope. Note that for shorttime periods the predicted system state � which bases on gyroscope data only � is veryaccurate and reliable.

7.3 The simpli�ed angle estimator in PCNNs (SAE-PC) 89

RF

PC

ref

+x=

−x+K

KF·(xFB−

−x)

xdiff = xFB − −x

KF update step result +x vs. increasing xdiff

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

×10−3

0

0.5

1

1.5

2

2.5

3

3.5

4

(a)

RF

PC

ref

KKF

xdiff = xFB − −x

Kalman gain KKF vs. increasing xdiff

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

×10−3

0

0.5

1

1.5

2

2.5

3

3.5

4

(b)

Abbildung 7.4: These two plots show the behavior of the update step of a 1D Kalman�lter in a PCNN compared to the behavior of a reference 1D Kalman�lter; the PCNN we use here is the same as the one we use for theSAE-PC implementation; the preferred value range is from −2 to 2,the number of neurons per population code is N = 51, the activationfunction parameter is KW = 200 and the width of the receptive �eld isRF = 0.1519


7.3.2 Performance of the estimator

Let's evaluate and compare the implementation of the SAE in PCNNs (SAE-PC). TheSAE-PC uses N = 51 neurons per population code and the preferred values of neuronsare linearly distributed in the range of −2 to 2. Consequently the distance between thepreferred values is 0.08. The SAE-PC uses only 2D hidden layers which have P = 2601hidden layer neurons. All the activation functions of PCNNs in the SAE-PC use thesame parameter KW = 200 which induces a receptive �eld width of RF = 0.1519.The hidden layer weight matrices Whl are stored in a sparse representation with athreshold value th = 0.001. The sparse representation of Whl,KF � which containslateral connection weights for the special Kalman �lter hidden layer � stores only 897 345lateral connection weights instead of 26012 = 6 765 201 (compression rate is 86.74%).The sparse representation of Whl, stab � which contains lateral connection weights forstabilizing hidden layers � stores only 90 993 lateral weights instead of 26012 = 6 765 201(compression rate is 98.65%). We notice that the sparse representation of hidden layerweight matrices saves a lot of memory. The input weight matrices Wpc for populationcodes in sub-layers of the output layer are usually far smaller than matrices Whl andcontain at most only P ·N = 132 651 connection weights.We summarize that the SAE-PC algorithm uses only 51 neurons to encode a scalar valueinto an activity, that the dynamic system of the Kalman �lter uses only 2601 hiddenlayer neurons with 897 345 lateral connections, that the stabilizing dynamic system needsonly 90 993 lateral connections in the hidden layer and that a scalar arbitrary functionwith a 2D input needs at most 132 651 connections to a hidden layer neurons.Although that the SAE-PC is only implemented in slow hybrid MATLAB and C-code,the implementation is quite fast and processes data almost in real time. The SAE-PCimplementation � with the settings from above � needs about dt = 27 ms per iterationstep (37 Hz). A SAE-PC with N = 31 neurons per population code needs about dt =10 ms per iteration step (100 Hz) and we are already very close to the sensor sample rateof 200 Hz. Note that tests have shown that there is almost no performance di�erencebetween a SAE-PC with N = 31 and a SAE-PC with N = 51 neurons per populationcode.It is clear that up to now I cannot implement the SAE-PC algorithm onboard a quadrotoras the algorithm is far too complex for ordinary embedded systems. Nevertheless theSAE-PC algorithm uses only simple calculations (additions and multiplications) andmost of these calculations can be done in parallel. Consequently the SAE-PC shouldbene�t when implemented on massively parallel calculating systems like the Spinnakerand an implementation in embedded system should become feasible.Let's have a look at �gure 7.5 and we see right a way that the SAE-PC works. Atall times the results of the SAE-PC stay close to reference values of the ground truthsystem (GT). If we compare �gures 7.5a and 7.5b to �gures 7.3a and 7.3b we see thatthe SAE-PC is only insigni�cantly inferior to the SAE with respect to roll angles andthat the SAE-PC is even slightly better than the SAE algorithm with respect to pitchangles. Indeed the SAE-PC delivers usable results with quite respectable average errorsof about 1 degree. In �gure 7.6 we get an overview of the performance of all attitudeestimators and in �gure 7.7 we have a zoom-in and look at attitude estimator results


from t = 40 s to t = 50 s.

SAE-PC

GT

φis

rollangle

inrad

t in sec.


10 20 30 40 50 60 70

−0.2

−0.1

0

0.1

0.2

(a)

SAE-PC

GT

θis

pitch

angle

inrad

t in sec.


10 20 30 40 50 60 70

−0.2

−0.1

0

0.1

0.2

(b)

Abbildung 7.5: Evaluate and compare the simpli�ed angle estimator implemented in aPCNN (SAE-PC) to ground truth data (GT); we deduced in subsec-tion 3.2.5 that the ground truth system reports erroneous pitch anglesbetween t = 18 s and t = 32 s


SAE-PC: φerr, rms = 0.9565 deg

SAE: φerr, rms = 0.8739 deg

ZAE: φerr, rms = 0.8271 deg

gyro: φerr, rms = 1.7465 deg

EKF-QR: φerr, rms = 0.5929 deg

GT

φis

rollangle

inrad

t in sec.


10 20 30 40 50 60 70

−0.2

−0.1

0

0.1

0.2

0.3

(a)

SAE-PC: θerr, rms = 1.1265 deg

SAE: θerr, rms = 1.1417 deg

ZAE: θerr, rms = 0.9565 deg

gyro: θerr, rms = 1.3013 deg

EKF-QR: θerr, rms = 1.0231 deg

GT

θis

pitch

angle

inrad

t in sec.


10 20 30 40 50 60 70

−0.2

−0.1

0

0.1

0.2

0.3

(b)

Abbildung 7.6: Comparison of results of di�erent roll and pitch angle estimators toground truth data (GT); the red line (gyro) is the integrated angularvelocity of the gyroscope; we can see that roll and pitch angles of thegyro drift over time while the angle estimators which fuse data of mul-tiple sensors deliver roll and pitch angles without drift; the EKF of thequadrotor, the z-axis estimator (ZAE), the simpli�ed angle estimator(SAE) and the implementation of the SAE in PCNNs (SAE-PC) cal-culate roll and pitch angles which are very close to ground truth data;we deduced in subsection 3.2.5 that the ground truth system reportserroneous pitch angles between t = 18 s and t = 32 s


SAE-PC: φerr, rms = 0.9680 deg

SAE: φerr, rms = 0.7424 deg

ZAE: φerr, rms = 0.6799 deg

gyro: φerr, rms = 1.6980 deg

EKF-QR: φerr, rms = 0.6580 deg

GT

φis

rollangle

inrad

t in sec.


40 41 42 43 44 45 46 47 48 49 50

−0.1

−0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

(a)

SAE-PC: θerr, rms = 0.9162 deg

SAE: θerr, rms = 1.0779 deg

ZAE: θerr, rms = 0.7922 deg

gyro: θerr, rms = 0.8382 deg

EKF-QR: θerr, rms = 0.9669 deg

GT

θis

pitch

angle

inrad

t in sec.


40 41 42 43 44 45 46 47 48 49 50

−0.1

−0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

(b)

Abbildung 7.7: This plot is a zoom-in into the plots of �gure 7.6a and 7.6b for a timeinterval of t = 40 s to t = 50 s; note that in this time interval the pitchangle of the SAE-PC is even better than the one of the EKF-QR; ne-vertheless for example at t = 45 s and t = 49.5 s we have bigger distancesbetween the pitch angle of the SAE-PC and the GT data; these are ti-mes where we have big linear accelerations while the pitch angle almostdoesn't change and thus all models based on ZAE have problems be-cause of the deteriorating feedback of the accelerometer; see subsection7.1.2 for additional information

8 Conclusions

In chapter 2 I have introduced a toolset which enables us to acquire sensor data of a�ying quadrotor and in chapter 3 I have evaluated this sensor data by comparing it toground truth data of the optical tracking system. The evaluation of quadrotor sensordata has shown that accelerometer measurements are noisy and measurements alongthe z-axis of the accelerometer are erroneous. Gyroscope measurements are very precisebut drift over time. Magnetometer measurements are not usable because high currentsdisturb the local magnetic �eld. The EKF of the quadrotor (EKF-QR) estimates rolland pitch angles with a very good performance but yaw angles only with poor perfor-mance. I concluded that the EKF-QR attitude estimator relies for yaw angles to muchon magnetometer measurements.In chapter 6 I have described in detail how I implement the simpli�ed angle estimator(SAE) of chapter 5 in PCNNs (SAE-PC) using the generic framework for PCNNs ofchapter 4. In chapter 6 I have also determined how PCNN parameters de�ne the Kal-man gain for 1D Kalman �lters.Finally in chapter 7 I have evaluated and compared the ZAE the SAE and the SAE-PCattitude estimator algorithms. The evaluation shows that SAE-PC works with noisy sen-sor data in a real world application. So I can immediately conclude that other PCNNsimplementation have a good chance to work too and that PCNNs can be used in a ge-neric manner for the implementation of fusion models. The performance of the SAE-PCis reasonable and the mean angle error is 1 degree or less.We have also observed that the receptive �eld width and the di�erence between pre-diction and feedback in�uences the Kalman gain in an PCNNs implementation. For bigdi�erences between prediction and feedback the Kalman �lter in PCNNs neglects thefeedback more and more. This might have an positive e�ect on the robustness of theSAE-PC. Despite that the SAE-PC is implemented in slow hybrid MATLAB and Ccode the algorithm processes data almost in real-time. All the calculations the SAE-PCuses are simple additions and multiplications and most of the calculations can be donein parallel. So the SAE-PC will bene�t with a higher processing speed when implemen-ted on massively parallel computing systems like SpiNNaker. However at the currentstate the SAE-PC algorithm is far to complex for low power embedded systems � notbecause of complicated operations but because of the amount of simple calculations thealgorithm needs.There are still many things which have to be investigated with regard to PCNNs. Theprocessing speed for a plain C-code implementation and for a SpiNNaker implementa-tion should be evaluated, the SAE-PC should be embedded to the quadrotor and betested in the control loop, the relationship between count of neurons and performanceof the algorithm should be determined, and the relationship between sensory gains andKalman gains for Kalman �lters with more than one dimension should be investigated.

96 8 Conclusions

It would also be interesting to know how di�erent receptive �eld widths � and thus di�e-rent regions where the Kalman gain changes � in�uence the performance of the SAE-PCalgorithm.

Literatur

[1] The px4 autopilot project, Internet: https://pixhawk.org/, Juni 2014.

[2] The qgroundcontrol program, Internet: http://qgroundcontrol.org/start, Juni2014.

[3] The mavlink protocol, Internet: http://qgroundcontrol.org/mavlink/start,Juni 2014.

[4] The naturalpoint inc., an optical tracker system manufacturer, Internet: https://www.naturalpoint.com, Juni 2014.

[5] The wireshark program, Internet: http://www.wireshark.org/download.html,Juni 2014.

[6] P. E. Latham, �Computing with population codes�, Okinawa Computational Neu-

roscience Course, S. 1�10, Nov. 2004.

[7] P. E. Latham, S. Deneve und A. Pouget, �Optimal computation with attractornetworks�, Journal of Physiology, Bd. 97, S. 683�694, 2003.

[8] S. Deneve, P. E. Latham und A. Pouget, �Reading population codes: a neuralimplementation of ideal observers�, Nature neuroscience, Bd. 2, Nr. 8, S. 740�745,Aug. 1999.

[9] A. Pouget, P. Dayan und R. Zemel, �Information processing with population co-des�, Nature reviews neuroscience, Bd. 1, S. 125�132, Nov. 2000.

[10] A. Pouget und L. H. Snyder, �Computational approaches to sensorimotor trans-formations�, Nature neuroscience supplement, Bd. 3, S. 1192�1198, Nov. 2000.

[11] S. Deneve, P. Latham und A. Pouget, �E�cient computation and cue integrationwith noisy population codes�, Nature Neuroscience, Bd. 4, Nr. 8, S. 826�831, Aug.2001.

[12] A. Pouget und T. J. Sejnowski, �Spatial transformations in the parietal cortexusing basis functions�, Journal of Cognitive Neuroscience, Bd. 9, Nr. 2, S. 222�237, März 1997.

[13] S. Deneve, J.-R. Duhamel und A. Pouget, �Optimal sensorimotor integration in re-current cortical networks: a neural implementation of kalman �lters�, The Journalof Neuroscience, Bd. 27, Nr. 21, S. 5744�5756, Mai 2007.

[14] P. R. MacNeilage, N. Ganesan und D. E. Angelaki, �Computational approaches tospatial orientation: from transfer functions to dynamic bayesian inference�, Journalof Neurophysiology, Bd. 100, S. 2981�2996, Okt. 2008.

[15] H. Lim, J. Park, D. Lee und H. J. Kim, �Build your own quadrotor�, IEEE Robotics

and an automation magazine, S. 33�45, Sep. 2012.

https://pixhawk.org/

http://qgroundcontrol.org/start

http://qgroundcontrol.org/mavlink/start

https://www.naturalpoint.com

https://www.naturalpoint.com

http://www.wireshark.org/download.html

98 Literatur

[16] H. Durrant-Whyte und T. C. Henderson, �Multisensor data fusion�, in Springer

Handbook of Robotics, B. Sciliano und O. Khatib, Hrsg., Springer-Verlag BerlinHeidelberg, 2008, Kap. 25, S. 585�612.

[17] R. Mahony, T. Hamel und J.-M. P�imlin, �Nonlinear complementary �lters onthe special orthogonal group�, IEEE Transactions on Automatic Control, Bd. 53,Nr. 5, S. 1203�1218, Juni 2008.

[18] S. Bonnabel, P. Martin und P. Rouchon, �Non-linear symmetry-preserving ob-servers on lie groups�, IEEE Transactions on Automatic Control, Bd. 54, Nr. 7,S. 1709�1713, Juli 2009.

[19] J. Laurens und J. Droulez, �Bayesian processing of vestibular information�, Bio-logical Cybernetics, Bd. 96, S. 389�404, 2007.

[20] J. K. Lee, E. J. Park und S. N. Robinovich, �Estimation of attitude and externalacceleration using inertial sensor measurement during various dynamic conditions�,IEEE Transactions on Instrumentation and Measurement, Bd. 61, Nr. 8, S. 2262�2273, Aug. 2012.

Cortically Inspired Sensor Fusion for Quadrotor Attitude...

Documents

Transcript of Cortically Inspired Sensor Fusion for Quadrotor Attitude...