FYS-STK3155/4155 Machine Learning - Project 3Classifying Pulsar Candidates With Machine Learning Methods

N. Obody(Dated: September 11, 2019)

The project deals with the classification of pulsar candidates using statistical measures of thepulsar candidates’ integrated pulse profile and the dispersion measure vs. signal-to-noise ratiocurve. The classification is performed using both neural networks and support vector machines.Both methods are shown to produce excellent results with a highest accuracy score of 98%, providedreasonable hyperparameters.

All material for project 3 may be found at:https://github.com/xxx/machinelearning20yy/tree/master/Project3

I. INTRODUCTION

In the field of radio astronomy, pulsars are amongthe most studied phenomena in nature. But despite as-tronomers’ long history with puslars, little is actuallyknown with certainty. However, much of the uncertaintylikely boils down to the difficulty of confirming pulsarobservations. While pulsars radiate unmistakable radiosignals, they are often lost in the sheer number of radiosignals observed by radio telescopes every day. Further-more, due to the uniqueness of pulsar radio signals, classi-fying pulsars in large data sets of radio observations havehistorically been very difficult as human supervision hasbeen a necessity. However, recent advancements in ma-chine learning and data mining has made this task muchsimpler by introducing incredibly fast, in comparison tohumans that is, classification methods. The aim of thesemachine learning methods is not to remove the role of theastronomer, but rather reduce large amounts of data tosmaller, manageable data sets. If done effectively, suchmethods can be used a primary tool for extracting likelypulsar-candidates in order to survey large regions of thenight sky in a fraction of the time.

II. THEORY

A. Radio Analysis of Pulsars

The following theory is mostly based on F.G. Smith’s1977 Pulsars [? ], with additional information taken from“Chapter 6: Pulsars” of Jim Condon’s online book Es-sential Radio Astronomy [? ].

1. The basics of pulsars

Pulsars are heavily magnetised rotating astronomicalobjects that emit regular pulses of electromagnetic (EM)radiation. Initially discovered in 1967 by Antony Hewishand Jocelyn Bell Burnell, it is generally believed by most

modern astronomers that pulsars are neutron stars, theproduct of some supernovae. There are several mechan-ims that result in the electromagnetic radiation (althoughstill not entirely understood), astronomers have thus de-vised three major categories of pulsars according to thesource of radiation: rotation-powered pulsars, accretion-powered pulsars and magnetars. Rotation-powered pul-sars use the loss of rotational energy, accretion-poweredpulsars exploit the gravitational potential energy of ac-creted matter, while magnetars are believed to powerits radiation pulses by the decay of an extremely strongmagnetic field (∼ 1010 T). Other than regular pulses ofradiation, the properties of the radiation from the dif-ferent categories are mostly different. Rotation-poweredpulsars radiate mostly in the Radio spectrum, while theaccretion-powered pulsars radiate mostly in the X-Rayspectrum. Magnetars have been observed to radiate bothin the X-Ray and gamma-ray spectra.

The data studied in this project stem from radio-telescope observations, thus only rotation-powered pul-sars are considered here.

2. ISM dispersion

As the name implies, radio pulsars are particular pul-sars that radiate in the radio spectrum. When as-tronomers first began studying radio pulsars they no-ticed that radiation from the same pulsar with differ-ent frequencies arrived on Earth at different times. Itwas quickly pointed out that this EM dispersion coin-cided with the expected dispersion of EM waves travel-ling through the cold plasma of the interstellar medium(ISM). Travelling through an ionised gas, the group andphase velocities of radio waves, vg and vp respectively,are related by

vgvp = c2 (1)

where c is the speed of light in a vacuum. The fractiveindex of the cold plasma is

µ =

√1−

(νpν

)2(2)

2

where ν is the radio wave frequency and νp is the plasmafrequency given by:

νp =

√e2neπme

(3)

Here, ne and me are the electron number density andme is the electron mass. Now, radio waves with ν ≤ νpare unable to propegate through the ISM. On the otherhand, waves with ν > νp do propegate through the ISMwith a group velocity of

vg = µc ∼=(

1−ν2p2ν2

)c (4)

Hence, the travel time T spent by a radio pulse from apulsar a distance L away is

T =

∫ L

0

1

vgdl =

1

c

∫ L

0

(1 +

ν2p2ν

)dl

=d

c+

(e2

2πmec

)ν−2

∫ L

0

ne dl (5)

The integral in (5) is commonly called the DispersionMeasure, and is commonly given in units of pc cm−3:

DM =

∫ L

0

ne dl (6)

The Dispersion Measure is not a trivial quantity to es-timate because it depends on the constituents and theirarrangements of the ISM, which in general is not uni-form. Despite accurate theoretical predictions of DM, itcontinues to be an essential variable parameter in radioastronomy analysis, more or this below. In astronomy-convenient units, the dispersion delay t = T −d/c due tothe ISM is

t ∼= 4.149 · 103ν−2 DM seconds (7)

where ν in MHz and DM in pc cm−3.

3. The integrated pulse profile

Due to difficulties with time-resolution, limited band-width, etc., studying a single radio pulse is considerablymore difficult than to study the “average pulse” acrossseveral frequencies. Taking into account the time delaydue to dispersion in the ISM (equation (7)), the radiopulses from radio pulsars may be folded to yield an “in-tegrated pulse profile” (IPP). Seeing that the DM is es-sential in the computation of the time delay, astronomersoften study the IPP as a function of DM. An example ofthe folding process is shown in figure 1, note that thedelay folds across several pulse periods. Moreover, whilethe time-signal is fairly chaotic, the integrated pulse pro-file is very focused.

Figure 1: The pulse profile of a radio wave experiencesa time delay due to dispersion in the ISM. The

integrated pulse profile is a folded profile that finds anaverage pulse profile by taking into account the time

delay. Source: [? ]

While all pulsars share the property of a stable pulseprofile, it turns out that each pulsar exhibits a particularprofile that is more or less unique. This unique profile,often nicknamed the “fingerprint” of the pulsar, can haveseveral peaks and be less or more spread out. Nonethe-less, some statistical properties such as a single peak, adouble peak, various minor peaks, etc., do occur quitefrequently.

4. Signal-to-noise ratio

An important quantity in observational astronomy isthe Signal-to-Noise Ratio (SNR), which in its simplestform is defined as the ratio of the power of the signal tothe power of the noise:

SNR =Psignal

Pnoise(8)

SNR is a measure of the quality of observation, or“amount of meaningful information” in an observation.Astronomers actually use the SNR to study the DM pa-rameter space, as selecting the optimal DM is essential forconducting a proper analysis of the IPP. Enter the theDM-SNR curve: the SNR score of an integrated pulse

3

profile as a function of DM. The DM-SNR curve is usu-ally flat for non-pulsating signals, however peaks aboutthe optimal DM if the signal does pulsate. Consequently,the DM-SNR curve is commonly used as a preliminarytest for identifying signals from pulsars in large data setsof radio signals.

B. Moments In Statistics

Given some continuous function f : R → R, its nth

moment µn(c) about the real value c is defined as

µn(c) =

∫ ∞−∞

(x− c)nf(x) dx (9)

Moments are used in several areas of science and math-ematics, most prominently in mechanics and probabil-ity theory. If f is a probability density (or strictly non-negative and normalizable),1 there exists two major no-tation conventions:

µn(c) = E[(X − c)n

]and µn(c) = 〈(x− c)n〉

The former is favoured by statisticians, while the latteris favoured by physicists, this project will employ thephysicists’ convention.

The moments most often studied in probability theoryare the so-called central moments µn:

µn = µn(µ) = 〈(x− µ)n〉 =

∫ ∞−∞

(x− µ)nf(x) dx (10)

where µ = 〈x〉 is the mean value of the distribution andµn = µn(µ) is introduced to simplify notation. The 0th

and 1st central moments are 1 and 0 respectively as f(x)is normalized and (x − µ)f(x) is centered about µ. The2nd central moment is the variance:

µ2 =⟨(x− µ)2

⟩= σ2 (11)

where σ is the standard deviation.Furthermore, the central moments are used to define

the normalized moments µn:

µn =µnσn

=〈(x− µ)n〉〈(x− µ)2〉n/2

(12)

Normalized moments are dimensionless quantities thatdescribe the underlying distribution independently of thescale of the data. The 3rd normalized moment is knownas the skewness of a distribution and quantifies the extent

1 A probability density f : D → R satifies

0 ≤ f(x), ∀x ∈ D and

∫Df(x) dx = 1.

Note that D is usually the entire real number line.

to which a distribution is skewed to “left or right” ofthe mean. Positive skewness implies the distribution isskewed towards x values greater than µ and vice versa.The 4th normalized moment is called kurtosis and is ameasure of how quickly the tails of a distribution tendsto zero, i.e. a measure of “peakness”. Kurtosis is usuallyreported as excess kurtosis, which is the kurtosis of adistribution relative to the normal distribution (whosekurotsis is 3). Hence the definition:

Excess Kurtosis = Kurtosis− 3 (13)

C. Machine Learning - Artificial Neural Networks

Artificial Nerual Networks (ANN or simply NN) is aSupervised Machine Learning (SML) technique that isinspired by biological neural networks in animal brains.Rather than a specific technique or algorithm, ANN is anumbrella term for several NN structures and algorithmswhose complexity, computational difficulty, etc., variesgreatly between different techniques.

1. The Multi-Layered Perceptron

This project will employ the so-called Multi-LayeredPerceptron (MLP) model, which is a particular ANN thatis based around the idea of organizing the network’s neu-rons, or nodes, in layers. A node is loosely based on theconcept of biological neuron, which is capable of reciev-ing an electrical signal, process it, and transmit a newsignal to other neurons. In the MLP model the neuronsare arranged in L consecutive layers with Nl number ofnodes per layer (unqeual number of nodes per layer isalloweed). There are three types of layers: input, hiddenand output layers. The input and output layers man-age the network’s inputs and outputs, while the hiddenlayers serve as the network’s data-processing backbone.Particular to the MLP, every node in each hidden layer isconnected to every node in the previous and the next lay-ers, but nodes within the same layer are not connected.The MLP structure is illustrated in figure 2, note theinclusion of “bias”: Bias is an additional property of alllayers except the input layer whose purpose is essentiallyto center the data about 0. Conceptually, one may thinkof bias as the role of the intercept in Linear Regressionanalysis.

So far the network has only been described using fluidterms such as “transfer of information” and “node con-nections”, this will be expanded upon now. The numberof nodes in the input layer N1 must exactly match thedimensionality of the input data, thus there is a one-to-one correspondence between the nodes and the indicesof the input data (usually a vector quantity). Similarly,the number of nodes in output layer NL must exactlymatch the output data of the network, again with a one-to-one correspondence between nodes and data indices.

4

previous layer

current bias

current layer

next bias

next layer

z1l

z2l

zl+11

zl+12

zl+13

zl−11

zl−12

zl−13

bl+1bl

Figure 2: An illustration of the structure of an MLPANN: Each node is connected to every node in the

previous layer and the next layer. The inputs to thecurrent layer consists of the data from the previous

layer (green connections) and the current layer’s bias(left blue connections). The outputs of the current layer

(red connections) together with the next layer’s bias(right blue connections) become the inputs of the next

layer, and so on.

It follows that in practice it is really only the hiddenlayers that may have a variable number of nodes Nl,l ∈ {2, . . . , L − 1}. The network processes input datalayer-by-layer, meaning each hidden layer processes thedata in sequence. Data from the input layer is sent tothe first hidden layer, processed, then transfered to thenext layer, and so on.

Mathematically speaking, the connection between twonodes i in layer l and j in layer l + 1 is a scaling opera-tion. The connection’s scaling factor is usually called theweight and is commonly denoted by wl+1

ij , note the con-vention of labeling a connection weight between layers land l+1 by l+1. Each node is connected to every node inthe previous layer, thus the purpose of weighting the datais to regulate the importance of each node in the previouslayer. However, note that unless some other operation isconducted on the data the network’s overall operationon the input data is only linear (meaning methods suchas Linear Regression are more appropriate). This is cir-cumvented by the addition of a non-linear operation onthe every node’s input, commonly called the node’s ac-tivation function. Common activation functions include:

logistic(x) =1

1 + e−x(14a)

sigmoid(x) = tanh(x) (14b)

rectifier(x) = max(0, x) (14c)

Other activation functions are also used. Technically, anetwork may have a different activation for each node,

but this is rarely done in practice. On the other hand,layer-wide activation functions are often implemented, somuch so that the following theory will base the mathe-matics on the presumption of layer-wide activation.

The data in node i in layer l is denoted by yli. Theconnection weight between node i in layer l and node jin layer l+ 1 is denoted by wl+1

ij . The accumulated datainput from all nodes in layer l received by node j in layerl + 1, adjusted for bias bj , is given by

zl+1j =

Nl∑i=1

wl+1ij yli + bli (15)

The input data received by the node is fl+1(zl+1j ), where

fl+1 is the activation function of layer l + 1. Hence, thedata in node j in layer l + 1 is given by

yl+1j = fl+1(zl+1

j ) = fl+1

(Nl∑i=1

wl+1ij yli + bl+1

j

)(16)

Arranging the node data in vectors yl, the node inputs invectors zl+1, the connection weights in a weight matrixW l+1 and the bias in bias vectors bl+1, then (16) can bereformulated as a layer-wide (matrix) operation:

yl+1 = fl+1(zl+1) = fl+1

(W l+1zl + bl+1

)(17)

To conclude this section: The answer to the question”What is a Multi-Layered Perceptron Neural Network?”is the following function:

y = fL

(WLfL−1

(· · ·(W 1x + b1

)· · ·)

+ bL)

(18)

where x = y1 and y = yL are the network’s inputs andoutputs.

2. Training an MLP

The basis of the MLP, and many similar network struc-tures, is the so-called back-propagation algorithm (BPA).The MLP is an example of a supervised machine learn-ing technique, meaning the desired outputs are known.In case of the MLP, it is the BPA that introduces super-vision.

At its core, the BPA is essentially a variant of gra-dient descent techniques. The basic problem of gradi-ent descent methods is to find the input xmin ∈ D thatminimises some function F : D → R. The algorithmis based on producing sucessive approximations of xmin,each closer to the true minium. Seeing that the goal isto minimise F , the direction of each step is chosen oppo-site of the gradient of F . The fundamental algorithm issummarised below:

5

Algorithm 1 Gradient Descent

1: Select a trial input x0.2: for i = 1, . . . , N :

3: Compute: xi = xi−1 − γi∂F

∂x

∣∣∣∣xi−1

4: end for

Note that the derivative is not scalar. The step length γnmay be variable or constant depending on the implemen-tation, although proper convergence is impossible withtoo large steps.

The F function to minimise in the BPA is the so-calledcost function, denoted C. Similar to activation functions,different cost functions are used in different problems.The cost function most used in classification analysis isthe so-called cross-entropy:

C(y|t) = −[tT ln(y) + (1− t)T ln(1− y)

](19)

where y is a vector of the network’s outputs and t is avector of the targets, i.e. the desired network output.2

Note that C(y|t) is univariate: it is a function of out-puts y provided the targets t (that is, t is not a variablequantity). The derivative of C(y|t) is

∂C

∂y= (y − t)�

(y ◦ (1− y)

)(20)

where � and ◦ denotes the Hadamard division and prod-uct (element-wise operations). Now, if this was a problemfor gradient descent methods, then y would be updateddirectly using Algorithm 1 and equation (20). However, yis produced by the network according to equation (18),which is dependent on the weights and biases of eachlayer. Hence, in order for the BPA to improve the net-work it must study C(y|t) with respect to all weights andbiases.

Although seemingly difficult, the problem is actuallysomewhat straight-forward, it all comes down to cleveruse of the chain rule and limiting the number of necessarycomputations. Observe that the derivative of C(y|t) withrespect to layer l can be written as

∂C

∂W l=∂C

∂yl∂yl

∂zl∂zl

∂W l(21a)

∂C

∂bl=∂C

∂yl∂yl

∂zl∂zl

∂bl(21b)

Now define the signal error of layer l as

δl ≡ ∂C

∂zl=∂C

∂yl∂yl

∂zl=∂C

∂yl◦ f ′l (zl) (22)

2 The introduction of t in the cost function is the actual intro-duction of machine learning supervision in MLP ANNs. Also,note that naming t the “targets” refers to the goal of neural net-works: namely to construct the network such that it reproducesthe target outputs.

where f ′l (zl) is the derivative of the activation function

of layer l. Clearly then equations (21) may be written as

∂C

∂W l= δlyl−1 (23a)

∂C

∂bl= δl (23b)

where zl = W lyl−1 + bl was used to evalute ∂zl/∂W l

and ∂zl/∂bl. The next step is to relate the signal errorof layer l to l+1, the idea to rewrite ∂C/∂yl in (22) usinga reverse chain rule:3

∂C

∂yl=

∂C

∂zl+1

∂zl+1

∂yl=[(δl+1

)TW l+1

]T=(W l+1

)Tδl+1

That is, the signal error of layer l is related to the signalerror of layer l + 1 via

δl =(W l+1

)Tδl+1 ◦ f ′l (zl) (24)

Equations (23), (22) and (24) are what constitutes thefour equations of the back-propagation algorithm. Theequations essentially boil down to the equivalent of step3 in Algorithm 1 (in particular (23)). Moreover, notethat the “back-propagated quantity” in the BPA is thesignal error, i.e., the network’s output error is propagatedbackwards through the network (thereby expressing thedependency of the cost function on the signal error of thedifferent layers).

The back-propagation algorithm is composed of 3steps: the feed forward cycle, the back-propagation andthe network update. The feed forward step refers to thecomputation of the network’s output; the input data x is“fed forward” to the output. The back-propagation stepinvolves the computation of the signal errors and the finalnetwork update is the updating of the network’s weightsand biases using the derivatives of the cost function. Thefull algorithm is shown below.

Algorithm 2 The Back-Propagation Algorithm

1: Initiate the network weights and biases.2: for i = 1, . . . , N :3: Feed forward input x.4: Compute δL.5: for l = L− 1, . . . , 1:6: Compute δl.7: end for8: Update the network weights and biases according to

W lnext = W l

prev − γi∂C

∂W lprev

blnext = blprev − γi∂C

∂blprev

9: end for

3 The awkward double-transpose notation is used to preserve thecorrect dimensions of ∂C/∂yl.

6

3. MLP Classifiers

The MLP network described up until now technicallydoes not perform a classification, but a regression. This isnot a problem however as adjusting the network to func-tion as a classifier is not particularly difficult. The onlymeaningful difference is in the output layer, in particularthe output layer activation function.

The most common choice of ouput layer activation forclassification problems is the softmax function:

softmax(x, i) =exi∑Nj=1 e

xj

(25)

where x is a vector of N components. The reasonwhy the softmax is so popular is that it is normalized:∑Ni softmax(x, i) = 1, which is an important property

with regards to probability. The idea to use the MLPnetwork as a regression model to produce probabilities,and in turn classify the input data according to the mostlikely classes.

4. Improving BPA convergence

First and foremost, it is very difficult to predict howthe BPA behaves with respect to a contant learning rate.Hence, a guarantee of convergence is much more likely ifthe learning rate is adjusted during the iteration. For thetime being, no particular updating scheme seems to beuniversally optimal. The updating scheme used in thisproject is the so-called “adaptive scheme”, which updatesthe learning rate if the cost function is not improved aftern iterations. This project will update the learning rate if10 consecutive iterations fail to improve the cost via:

γn+1 =γn5

(26)

As it stands, notice that Algorithm 2 makes no men-tion of the possibility of multiple data inputs x. A simpleway to accomodate several data samples would be to it-erate over the available samples xi within the BPA loop.However, this is very error-prone as it is likely to favourthe first samples (especially with the adaptive learningrate). This can be slightly improved by using a stochas-tic approach, meaning the samples are picked at random.Nevertheless, this would only choose “the first samples”stochastically, not fix the underlying problem. A betterapproach is to use the average of several samples cho-sen at random, which means the mathematics must besomewhat adjusted (although not much): The input vec-tor x is replaced by an input matrix X =

[x1 · · · xM

],

which implies that the vector quantities δl, zl and yl nowbecome equivalent matrix quantities. Both W l and bl re-

main the same. Finally δl is reduced from a matrix ofcolumn vectors to the average column vector before beingused to compute ∂C/∂W l

prev and ∂C/∂blprev. What re-mains now is to select the set of input vectors to use in X.

Including all vectors usually leads to too large matrices,and is also sensitive to outliers. The so-called StochasticGradient Descent (SGD) method solves this by dividingthe training set (all samples) into K “mini-batches” ofsize B = N/K. The SGD update is:

xi = xi−1 − γiB∑j=1

∂F

∂x

∣∣∣∣x=xj

(27)

Yet another improvement of the standard gradient de-scent method is the so-called “Momentum Gradient De-scent” (MGD). The idea is to include an additional “mo-mentum” term whose purpose is to guide the descent inregions where the gradients are much larger in some di-mensions than others (which is common in regions closeto local minima). Because of MGDs innate relationshipwith mechanics, common practice is to denote the gradi-ent by vi. The MGD update is then written as

xi = xi−1 − ηivi−1 − γivi (28)

where ηivi−1 is the “momentum”.

D. Machine Learning - Support Vector Machines

One of the most versatile and diverse approaches toSML is Support Vector Machines (SVM). SVMs are verypopular for both regression and classification problems,this project however will only perform a binary classifi-cation. The mathematics are therefore presented with abinary labelling in mind.4

1. The basic principles behind SVMs

The basic idea of an SVM classifiers is to divide a dataset into discrete classes using a decision boundary, whosemathematical description is that of a hyperplane. Tech-nically speaking, hyperplanes are affine subspaces thatrepresent the direct generalisation of the concept of two-dimensional planes located in three-dimensional spaces.It happens that any decision boundary of a p-dimensionalpredictor space must be a (p−1)-dimensional hyperplane.

Hyperplanes may be parametrised via a linear equa-

tion using an intercept β0 and a parameter vector βT =(β1 · · · βp

). Written in set notation, the definition of a

hyperplane in a real p-dimensional space is

hyperplane = {x | xT β + β0 = 0} (29)

4 SVM classification with several classes is actually just a recursiveuse of binary classifications. E.g. Dividing the real number lineinto three classes (−∞, 0), [0, 10] and (10,∞) is the same asfirst dividing the real number line into (−∞, 0) and [0,∞), thensubdividing [0,∞) into [0, 10] and (10,∞).

7

This definition of a hyperplane is particularly useful, be-cause it accounts for the two regions separated by the

hyperplane by changing the equality in xT β + β0 to ei-ther < or >. Hence, a natural binary classification basedon (29) is to simply use the sign:5

class label = sign(xT β + β0) (30)

where the binary label (i.e. ±1) is an arbitrary choicethat simplifies notation. Note that the description ofdecision boundaries as hyperplanes necessarily restrictsthe possible boundaries to linear boundaries. It turns outthat this problem is easily circumvented by introducinga benefitial transformation of the predictor space. Thisis done in later sections.

Suppose a particular linear decision boundary is

parametrised by β0 and β. By definition, any point xi on

the hyperplane satisfies βTxi = −β0, and consequentlyany two points on the hyperplane x1 and x2 must there-

fore satisfy βT (x2 − x1) = 0. It follows that β/||β|| isa unit vector normal to the surface of the hyperplane.Now consider any point in the predictor space x: be-cause the hyperplane is infinite in reach, there must existsome point on the hyperplane x0 such that the distance

from x to x0 is minimal. Because β/||β|| is necessarilythe unit vector pointing along the line defined by x andx0,6 the signed distance from x0 to x must satisfy

βT

||β||(x− x0) =

1

||β||(βTx + β0) (31)

That is, the signed distance from any data point x to

x0 on the decision boundary is proportional to βTx +β0. The smallest distance between any data point andthe decision boundary is called the decision boundary’smargin M . The mission statement of SVMs is to find a

hyperplane (i.e. determine β and β0) that maximises themargin. Using the class labelling

yi =

{+1, xTi β + β0 ≥ 0

−1, xTi β + β0 < 0(32)

the SVM mission statement may be expressed as an op-timization problem:

yi

||β||(xTi β + β0) ≥M, i ∈ {1, . . . , N} (33)

5 This classification rule actually does not include points on thedecision boundary, thus common practice is to let these pointsbelong to the positive sign class by default.

6 This follows from the fact that the dimensionality of the hyper-plane is one less than the dimensionality of the predictor space.That is, the only degree of freedom that separates x from x0 isthe direction of the unit vector normal to the hyperplane surface,which is β/||β||.

or alternatively if ||β|| = 1/M :

minβ,β0

1

2||β|| subject to yi(x

Ti β + β0) ≥ 1, ∀i (34)

This is an example of a convex optimization problem, andis appropriately studied using the method of Lagrangemultipliers.

The Lagrangian primal to be minimized is

L(β, β0, λ) =1

2||β||2 −

N∑i=1

λi(yi(x

Ti β + β0)− 1

)(35)

Its derivatives are

∂L∂β

= β −N∑i=1

λiyixi (36a)

∂L∂β0

= −N∑i=1

λiyi (36b)

Setting ∂L/∂β = 0 and ∂L/∂β0 = 0, then inserting theresults back into (35) one finds the Wolfe dual (a maxi-mization problem):

L(λ) =

N∑i=1

λi −1

2

N∑i=1

N∑j=1

λiλiyiyjxTi xj (37a)

subject to λi ≥ 0 and

N∑i=1

λiyi = 0 (37b)

In addition to the constraints of the Wolfe dual, the solu-tion must also satisfy the so-called KKT (Karush-Kuhn-Tucker) conditions:

λi(yi(x

Ti β + β0)− 1

)= 0, ∀i (38)

It follows from (38) that either of two events is possible: If

λi > 0, then it must be so that yi(xT β+β0) = 1, in which

case xi is on the boundary. Else, if yi(xT β + β0) > 1,

then λi = 0 and xi is not on the boundary. Due to theparticular role of the xi vectors in the Wolfe dual, theyare often called the support vectors of the hyperplane(hence the naming of SVM). Computationally the Wolfedual optimization problem is actually much simpler thanthe Lagrangian primal optimization problem. Efficientalgorithms have been developed and are implemented inmost standard optimization software.

Having found the Lagrange multipliers λi, the decisionboundary hyperplane is found via

β =

N∑i=1

λiyixi and β0 =1

yi− βTxi, ∀i

The final SVM classifier is given by equation (30).

8

2. Soft classifiers

The classifier described in the previous section is whatis known as a hard linear classifier, i.e. a linear classi-fier that does not allow any cross-over between the twoclasses. Consequently, hard linear classifiers assume thatat least one hyperplane that successfully separates thedata set does exist, but this is easily not the case forall data sets. For example, any presence of significantnoise could easily produce some data points that “fallon the wrong side” of the decision boundary. Becauseof its shortcomings, most modern SVM implementationsdoes not use a hard classifier, but instead a so-called softclassifier. Soft classifiers are similar to hard classifiers intheir structure and purpose, however allows some slackwith regards to the data set.

Define the slack variables ξ =(ξ1 · · · ξN

), where ξi ≥

0, and modify equation (33) to “allow some slack” on theform:

yi

||β||(xTi β + β0) ≥M(1− ξi) (39)

or alternatively if ||β|| = 1/M :

minβ,β0

1

2||β|| subject to yi(x

Ti β+β0) ≥ 1−ξi, ∀i (40)

The total violation of (33) is now given by∑Ni=1 ξi, hence

bounding the sum from above by some constant C allowsfor ease of control of the total slack. Equation (40) canbe re-expressed in the equivalent form:

minβ,β0

{1

2||β||2 + C

N∑i=1

ξi

}(41a)

subject to ξi ≥ 0, yi(xTi β + β0) ≥ 1− ξi, ∀i (41b)

The Lagrangian primal to minimize is now:

L(β, β0, λ, γ) =1

2||β||2 + C

N∑i=1

ξi −N∑i=1

γiξis (42a)

−N∑i=1

λi(yi(x

Ti β + β0)− (1− ξi)

)(42b)

From here the derivation follows the same outline as thenon-slacked problem, the resulting Wolfe dual to maxi-mize is given by (37a) with the constraints

maxλL subject to 0 ≤ λi ≤ C and

N∑i=1

λiyi = 0 (43)

The corresponding KKT conditions are

λi(yi(x

Ti β + β0)− (1− ξi)

)= 0 (44a)

γiξi = 0 (44b)

yi(xTi β + β0)− (1− ξi) ≥ 0 (44c)

for all i. Resorting back to the classifier is identical tothe non-slacked case.

Table I: Commonly used classes of kernels in SVMclassification.

Class of Kernels K(x,y)

Linear functions xTy

nth degree polynomials (xTy + r)n

Radial Gaussians e−γ||x−y||2

Sigmoids tanh(xTy + r

)3. Non-linear classifiers

So far, only linear classifiers have been discussed. Itturns out that a simple “kernel trick” can be exploitedto produce non-linear classifiers. Consider the followingbasis transformation of the predictor space x to h(x):

h(x) =(h1(x) · · · hH(x)

)T(45)

where H is a positive integer (which can be larger thanN). The so-called kernel of h(x) is given by

K(xi,xj) = h(xi)Th(xj) (46)

Proceding through the linear classification as outlined inprevious sections one arrives on the Wolfe dual

L(λ) =

N∑i=1

λi −1

2

N∑i=1

N∑j=1

λiλiyiyjK(xi,xj) (47)

Note that (47) is not expressed in terms of the ba-sis expansion h(x), but instead of the kernel functionK(xi,xj). This is implies that a non-linear classificationrequires only knowledge of the desired kernel, and notthe underlying basis transformation. Common kernels inthe SVM literature are shown in table I.

To revert back to the original “hyperplane” (which isnow possibly a curved surface) one would necessarily re-quire an expression for the basis expansion h(x). How-ever, this can be avoided by introducing the function

f(x) = h(x)T β + β0 =

N∑i=1

λiyiK(x,xi) + β0 (48)

Hence, instead of (30), the binary classification of newdata points is given by

class label = sign(f(x)

)(49)

III. METHOD

The data set used in this project can be downloadedfrom the following Kaggle post:https://www.kaggle.com/pavanraj159/predicting-a-pulsar-star

The data set contains just under 18,000 samples of pulsarcondidates from the High Time Resolution Universe Sur-vey, all post-verified as either a known pulsar or a knownnon-pulsar. The data set originates from the 2012 article[? ] by S.D. Bathes et. al. in which an MLP classifierwas successfully used to “blindly detect 85% of pulsars”.

9

A. Preparation

The data set contains of 8 real-valued predictors and1 target label. The predictors include the mean value,standard deviation, skewness and excess kurtosis of thepulsar candidates’ IPP and DM-SNR curve. The targetlabel is a binary label ±1 that identifies the candidateas a pulsar (+1) or a non-pulsar (-1). A histogram foreach predictor has been included in appendix A, the pul-sars and non-pulsars have been separated to highlight thedifference.

Despite some minor overlap in the predictor space,there are clear differences between the pulsars and non-pulsars, and consequently an appropriate neural networkor SVM should not have much trouble separating the two.With this in mind, this project deals not with whether thetwo classes are separable, in fact Bathes et. al. proved asmuch in their original article, but instead to what extentare neural networks and SVMs applicable.

The machine learning in this project is implementedusing scikit-learn [? ], in particular the classesneural network.MLPClassifier and svm.SVC, in addi-tion to the functionality provided by train test splitand StandardScaler. The train-test-split function ran-domly divides a data set into a training set and test set.The scaler transforms the predictors space by standard-izing the predictors, i.e. remove the mean and scale thevariance:

Z =X − 〈x〉σX

(50)

The neural networks and SVMs are compared using aso-called accuracy score. This score is a measure of theaverage correct identification of a pulsar or non-pulsar:

accuracy =1

N

N∑i=1

Ii, Ii =

{1, correct label

0, incorrect label(51)

where i traverses the test set. The accuracy score is num-ber between 0 and 1.

B. NN Hyperparameter Analysis

The MLP network is very robust, converging on a min-imum for almost all reasonable choices of hyperparame-ters. Ideally one would be able to vary all hyperparam-eters simultaneously, but this is simply not feasible ona standard computer, and in fact, probably not neces-sary. Instead certain “types” of hyperparameters, e.g.“learning parameters” and “structure parameters”, willbe studied separately.

There are countless possible variations of even the sim-plest of networks. Therefore, the following restrictionsare introduced in order to reduce the number of combi-nations of paramaters:

1. All layers in a network have the same number ofnodes.

2. All layers in a network use the same activation func-tion.

3. With the exception of section III B 3, the pulsardata set is divided 50-50 into training and test sets.

Having trained a network, the performance of the net-work is measured according to three metrics: its accu-racy score, the number of required training epochs andthe time spent training. The accuracy score is interestingfor obvious reasons, the two latter metrics are interestingbecause they reveal the network’s training efficiency.

1. Network complexity

The first stage of the analysis takes a look at whethersimple or complex networks work best with the pulsardata set. Recall that L is the number of layers andN is the number of nodes per layer. To span a largerange of complexity levels, 900 networks structures areconsidered: The networks use 30 linearly spaced L val-ues between 1 and 50, and 30 linearly spaced N valuesbetween 1 and 100.7 Each network is trained using lo-gistic, sigmoid and rectifier activation. Hence, a total of3×30×30 = 2700 unique networks are trained. The net-work designs used in the following sections will be largelybased on the results of this complexity analysis.

2. Network learning scheme

Having looked at the network designs, this analysis willstudy the network’s learning phase. The hyperparame-ters of interest are the initial learning rate γ0 and thestrength of the learning momentum η. Note that equa-tion (28) allows for a varying momentum strength, butthis will not be explored here. This analysis will lookat 17 × 15 networks using 17 logarithmically spaced γ0values between 10−4 and 100, and 15 linearly spaced ηvalues between 0.01 and 0.99. Each network is trainedusing a logistic, sigmoid and rectifier activation, making atotal number of 765 unique learning schemes per networkstructure.

7 The “linearly spaced” L and N values are created usingnp.linspace(...).astype(int). The type switch from float

to int may slightly interfer with the linear spacing.

10

3. Overfitting the training data

The previous two analyses have studied hyperparam-eters concerning the network itself, this analysis will in-stead look at whether the networks are sensitive to dif-ferent divisions between the training and test sets. Eachnetwork is trained against 100 training sets with trainingset sizes between 10% and 90%.8

C. SVM Hyperparameter Analysis

While much harder to train than neural networks,SVMs are very flexible. Unfortunately, this flexibilitycomes at a price of large hyperparameter spaces, whichin general are very difficult to fully explore in depth. Likewith the neural networks, hyperparameters will insteadbe studied in groups of similar types of parameters.

This project will use the following assumptions on thebehaviour of the SVMs:

1. While different kernels may behave differently withrespect to the same violations strengths, all ker-nels are assumed to be similarly with respect to theconcept of violations. In particular, different ker-nels may disagree what value constitutes a “largeviolation”, but necessarily agree on what a largeviolation is.

2. The kernel hyperparameters are assumed to bemore or less invariant of the violation strength,meaning the kernel hyperparameters and the vi-olation strength can be explored separately.

3. Similar to the violation strength, the kernel hyper-parameters are also assumed to be more or less in-variant of a particular the training set size.

4. SVMs are particularly sensitive to uneven predic-tor spaces, thus all predictors are standardized inorder to avoid long runtimes (this does not affectthe results).

The assumptions presented above are true in general, butare introduced in order to limit the score of the project.

Unless otherwise specified, references to “SVMs”should be interpreted as 4 different support vector ma-chines: one with a linear kernel, one with a 3rd degreepolynomial kernel, one with a radial Gaussian kernel andone with a sigmoid kernel. The parameters of the ker-nels, unless otherwise specified, are set to the defaultparameters of sciki-learn. More details on the defaultvalues can be found here: https://scikit-learn.org/stable/modules/svm.html

8 By “training set size”, what is meant is the portion of the originaldata set that is selected as the training set.

Having trained an SVM, the performance of the SVMis measured according to two metrics: its accuracy scoreand training time. Measuring both the accuracy and thetraining time may yield a compromise between accuracyand efficiency.

1. Violation control

It is very important to correctly control an SVMs viola-tion in order to optimize results. The goal of this analysisis to perform a preliminary check of how different kernelsbehave with different violation strengths C. To this ex-tent, 100 logarithmically spaced C values between 10−5

and 102 are used to train the SVMs.

2. Overfitting the training data

Mirroring section III B 3, this analysis is concernedwith whether the SVMs are sensitive to different divisionsbetween the training and test sets. Again, the hyper-parameters concerning the kernels are also not exploredhere. The violation strength for the different kernels arechosen based on the results from the previous analysis.Each SVM is trained using 30 training sets with trainingset sizes between 5% and 95%.

3. Kernels

The aim of this section is to study the hyperparametersof the polynomial, radial Gaussian and sigmoid kernels(the linear kernel does not have a hyperparameter, andthus is not included). The violation strength for the dif-ferent kernels are chosen based on the results from theprevious analysis. For a description of the hyperparam-eters, see table I. There are no absolute guidelines fordetermining the hyperparameters, the values presentedhere are based on trail and error.

For the polynomial kernel, the hyperparameters are thepolynomial degree n and the center shift r. The polyno-mial degrees considered include 2,3,4,5 and 6. For eachpolynomial degree, 50 logarithmically spaced r values be-tween 10−4 and 102 are used.

In case of the radial Gaussian kernel, the hyperparame-ters include only the scaling factor γ. 200 logarithmicallyspaced γ values between 10−6 and 102 are used.

The hyperparameters of the sigmoid kernel consistsonly of an argument shift r. As tanh is an odd function, itis uneccessary to explore negative r values. Consequently50 r values between 10−6 and 106 are used.

11

IV. RESULTS

A. Neural Networks

1. Network complexity

The main results of the complexity analysis are shownin figure 3. Additional results including similar figureshave been included in the GitHub repository.

First and foremost, all networks score 90% or higher,something which is surprising considering Bathes et. al.’sscore of 85%. Second, there are clear differences betweennetworks with different activations, most notably the lo-gistic networks compared to the other two. Third, re-gardless of the activation function, the networks exhibita considerable boundary in the training times images,ca. along the 30 nodes per layer vertical. Interestinglyhowever, this boundary is not present in the number ofepochs image, meaning this boundary does not representa mathematical turning point within the BPA algorithm.It follows that this boundary most likely actually stemsfrom a change of numerical algorithm within either NumPyitself or the BPA implementation in scikit-learn.

Another surprising result is that the networks, regard-less of activation, seem to exhibit a boundary betweenhigh and low accuracy. The boundary is particularly in-teresting in the logistic case where upon closer inspection,the boundary has been shown to exclusively include net-works with a single layer or 2 layers.

As expected, there is a general trend for more com-plex networks to require more training, both in numberof epochs and time. Exempt from this rule however isthe rectifier networks, which require particular attentionalong the boundary between the high and low accuracyregions.

For the logistic networks, using 1 or 2 layers with about10-20 nodes is recommended. Such a network is trainedfast and scores optimally. Sigmoid networks on the otherhand are a little more flexible as it can accept as manyas about 25 layers without suffering from long training.However, the layers should have more than about 20nodes per layer in order to be optimally accurate. Fi-nally, the rectifier networks are optimized using a smallnumber of layers, say less than 10, with ca. 10 nodes ormore. Overall, the logistic networks train much fasterthan the sigmoid and rectifier networks. On the otherhand, the high accuracy regions of the logistic networksis much smaller than for the other two, making themprone to large drops in accuracy with only small adjust-ments to the networks.

In order to standardize the neural networks for thenext analyses, two different structures are implemented:one with a single layer and 8 nodes and one with 3 layers

and 30 nodes per layer. For lack of better terminology,the former will be referred to as the “simple” structure,while the latter is referred to as the “complex” structure.

2. Network training scheme

The main results of the learning scheme analysis areshown in figure 4. Additional results have been includedin the GitHub repository.

A striking difference between the simple and complexnetworks is the boundary between the low and high ac-curacy regions. Whereas the simple networks exhibit amore gradual boundary, the complex networks show amuch more intense boundary reminiscent of the bound-aries in figure 3.

Furthemore, the simple and complex networks also donot agree on the optimal learning parameters. The logis-tic networks exhibit the most stark contrast: while thesimple networks are optimized for a whole range of pa-rameters, the complex networks require more attentionto detail. The center of the high accuracy region for thesimple networks is just on the boundary the low and highaccuracy regions. Meanwhile, the center of the high accu-racy region for the complex networks is displaced down inthe parameter space. Similar to the logistic networks, therectifier networks also limit its range of optimal learningparameters, albeit to a lesser degree and the high accu-racy center remains intact. The sigmoid networks on theother hand exhibit the most minimal difference betweenthe simple and complex structures.

Seeing that the networks learn faster in the lower rightregions (generally speaking) of the images, it is no sur-prise that the training times are generally very small inthis region. There are some notable exceptions however,in particular for the logistic networks. The simple lo-gistc networks behave exactly as predicted with respectto training times and the number of epochs spent train-ing, however the complex networks form a centered bandof learning parameters (with γ0 ≈ 10−2) in which train-ing is very slow. The band is most likely linked to thehard boundary between the low and high accuracy re-gions as it seems to trace the boundary quite closely.

To strike a compromise between learning efficiency andaccuracy, most networks should be trained using a set oflearning parameters along the diagonal from the lowerleft to the upper right. There are, of course, exceptionsto this rule. In case the complex logistic networks aredesired, it is important to first survey for the approximatelocation of the low-to-high accuracy boundary. To userectifier networks, it is adviced to a slightly smaller initiallearning rate.

For the next analyses, the following three learning pa-rameter sets will be used: one with (γ0, η) = (10−1, 0.3),one with (γ0, η) = (10−2, 0.5) and the final with (γ0, η) =(10−3, 0.7).

12

Figure 3: Hyperparameter analysis of the complexity of neural networks. In general, network complexity increasesalong the diagonal from the upper left to the lower right. An upper threshold of 150 epochs and 200 seconds is used

to focus the colormaps, white pixels have passed the threshold.

13

(a) The simple networks. An upper threshold of 1000 epochs and 25 seconds are used to focus the colormap, white pixelshave passed the threshold.

(b) The complex networks.

Figure 4: Hyperparameter analysis of the learning scheme of neural networks. As a general rule, the learning rateincreases along the diagonal from the upper left to the lower right.

14

(a) Simple networks, trained with γ0 = 10−1 and η = 0.3. (b) Complex networks, trained with γ0 = 10−3 and η = 0.7.

Figure 5: Hyperparameter analysis of training set size’s impact on neural networks overfitting.

3. Training set overfitting

The overfitting analysis was run for all network ac-tivations, for both simple and complex network struc-tures, and for each of the three learning parameter setsfrom the training scheme analysis. Much of the resultshighlight the same relationships, as well as reiterate theresults of the previous analysis. To avoid uneccessaryfigures, only two plots have been included: the analysisusing simple networks, trained with (γ0, η) = (10−1, 0.3),and the analysis using complex networks, trained with(γ0, η) = (10−3, 0.7). The rest of the material, togetherwith some additional material using different networkstructures, is included in the GitHub repository. Themain results of the analysis are shown in figure 5.

Quite unexpectedly, the training set sizes have almostno impact on the network accuracies. Nevertheless, somedata results do show a slight decrease in accuracy withtraining set sizes less than 20%. In addition, some net-works’ accuracies become less stable with increased train-ing set sizes, e.g. the logistic networks in figure 5b. Thiscould suggest some minor overfitting, however the result-ing variations in the accuracy scores are miniscule com-pared the difference between the low and high accuracyregions seen in previous analyses.

Seeing that (γ0, η) = (10−1, 0.3) is just on the low-to-high accuracy boundary for the rectifier networks (seefigure 4), the unstable behaviour of the rectifier networksin figure 5b comes as no surprise. The erratic behaviourseems to be uniformly spread, suggesting whether anerual network falls within the low or accuracy regionsboils down to chance. This hypothesis is further sup-ported by the fact that the neural network’s weights andbiases have random initial conditions. There does notseem to be any clear mechanisms with which such a strictboundary should appear (as opposed to smooth bound-ary), other than its a peculiarity of the BPA algorithmitself.

Furthermore, there is a general trend for training timeto increase with increasing training set size, but this isexpected. Especially in the case of the simple networks,which are more or less the same for all network activa-tions. However, in the case of the complex networks, thesigmoid networks require a much larger numer of epochs,and thus time, to train. Despite this however, the sig-moid networks have no accuracy benefit over the rectifiernetworks (that is, in figure 5b).

It is clear that the training set size need not be greaterthan say 20%. As long as the training set is large enough,increasing it yields neither accuracy nor efficiency.

15

B. Support Vector Machines

1. Violation control

The results of the violation control analysis are shownin figure 6. While the SVMs with linear, polynomial andradial Gaussian kernels generally improve with increas-ing violation, the sigmoid SVM does not. Additionally,the simoid SVM is also the least accurate SVM overall,barely reaching an accuracy of above 91%. Furthermore,whereas the accuracy scores for the linear and polyno-mial SVMs improve with increasing violation, their cor-responding training times explodes. On the other hand,the radial Gaussian SVM is capable of achieving similaraccuracy with only a fraction of the training time.

Overall the linear kernel is the fastest SVM to train,in addition to having the highest score for all violationsgreater than 10−4. It is the clear better alternative incase a fast SVM is needed.

The radial Gaussian SVM are clearly optimized with aviolation of about 1: the accuracy is maximal (ca. 98%)as well as the training time is minimal (ca. 0.5 seconds).Interestingly, the training times for the sigmoid SVMdoes not seem to be affected by the increasing violation,despite its major drop in accuracy. The training timesfor the linear and polynomial SVMs are optimized for aviolation between 10−2 and 10−1, with which the SVMsscore just below 98%. No particular violation strengthseems to optimize the sigmoid SVM’s training time, al-though it is slightly decreasing with increasing violation.

2. Training set overfitting

The results of the training set size analysis are shown infigure 7. Interestingly, in accordance with the neural net-works, the SVMs also show little variation with the differ-ent training set sizes. The training times increases withincreasing training set size, as expected, although admit-edly much faster for the sigmoid SVMs. The same pat-tern of slightly larger fluctuations in the accuracy scoreswith increasing training set size can also be seen here.

Much like the neural networks, no rel benefit is addedby incresaing the training set size. Hence, a training setsize of more than about 20% is not necessary.

It’s important to note that the trend seen in figures 5and 7 is most likely due to the separability of the pulsardata set, and not the nerual networks or SVM kernels.

Figure 6: A crude analysis of how the different SVMkernels behave with respect to different violation

strengths.

Figure 7: A crude analysis of how the different SVMkernels behave with respect to different training set

sizes.

16

3. Kernel hyperparameters

The results for the SVM kernel hyperparameter anal-yses are shown in figures 8, 9 and 10. The three kernelsexhibit quite different behaviour with respect to their re-spective hyperparameters.

The accuracy of the polynomial kernels generally in-crease with increasing polynomial shift until about r ≈101, where most of the kernels exhibit some form of max-imum or peak accuracy (although the 2nd degree kernelcould still be on the rise, the details are somewhat un-clear). When it comes to training times, increasing thedegree of the polynomial has more or less a blatant neg-ative effect. Out of the different polynomial kernels, the3rd degree polynomial stands out as both very effective,yet highly accurate (which is lucky considering the poly-nomial kernel used in the two previous analyses were bothof degree 3).

Out of the different SVM kernels, the radial GaussianSVM is sepecial in that it features a very clear optimalhyperparameter, in fact a scaling factor of just aboutγ = 10−1. This scaling both maximizes accuracy andminimizes training time, with little to no cost.

On the other hand, the Sigmoid SVM accuracy ismore or less completely invariant of the argument shiftr. However, the training times make a hard drop aroundr ≈ 101. Hence, a large r, say r = 103, is clearly benefi-tial.

Figure 8: Hyperparameter analysis of the polynomialSVM kernel.

Figure 9: Hyperparameter analysis of the radialGaussian SVM kernel.

Figure 10: Hyperparameter analysis of the sigmoidSVM kernel.

17

V. DISCUSSION

The goal of this project is to examine whether neuralnetworks and support vector machines could be intro-duced as a standard tool for examining large data setsin radio astronomy. In particular whether these methodscan be used to extract pulsar candidates from radio sig-nals using their integrated pulse profiles and DM-SNRcurves. Bathes et. al.’s 2012 attempt yielded an MLPnetwork that was able to “blindly” detect 85% of pulsarcandidates. However, as the results of this project hasshown, the methods discussed in this project have thepotential to improve his accuracy to as high as 98%.

When it comes to the neural networks, it was clearthat less complex networks, i.e. networks with less thanabout 10 layers, generally performed just as well as themore complex networks. In addition, the smaller networkstructures required much less training in general. An in-tersting case was the logistic networks, which requiredeither a single layer or 2 layers in order to perform op-timally. Overall, all network activation functions yieldedgood results, if parametrised correctly.

Nonetheless, based on the variety of acceptable hyper-parameters that optimized the networks, the sigmoid net-works definitely stood out as the most easy to work with.Especially considering that their training times were notparticularly larger than the other two network activa-tions. The sigmoid networks also performed very wellfor almost all choices of learning parameters, somethingthat the logistic and rectifier networks did not. In fact,the sigmoid network is very stable for most combinationsof hyperparameters; performing not only adequately, butexceptionally well.

Provided reasonable violation strengths, the supportvector machines proved reasonble adversaries for thenerual networks; also scoring as high as 98%. This wasnot true for every kernel however, the sigmoid kernel wasnot even able to score 92%, which for comparison wasthe worst score for the other kernels.

Apart from the linear kernel, which was by far both thefastest and most accurate SVM, both the radial Gaussianand the 3rd degree polynomial kernels achieved an accu-racy score of 98% using less than a second to train. Thepolynomial SVM is trained slightly faster than the radialGaussian SVM, but this is only measureble for very large

training sets. Judging by the separability of the pulsardata set (see histograms in appendix A), the data set ismost likely linear in its boundary. However, the radialGaussian and polynomial SVMs are much more flexiblethan the linear SVM. In case a less linearly separabledata set is used, the linear SVM is discouraged.

In general, the SVMs have exhibited slightly longertraining times than the neural networks. Thus, neu-ral networks could be adventageous in case training effi-ciency is important. However, the neural networks tendto have many more hyperparamaters, making the hyper-parameter analysis that follows much more demandingthan for an SVM. Take for instance the radial Gaus-sian SVM whose only hyperparameters are the violationstrength and the scaling factor, and for comparison, asigmoid neural network, which requires both a networkstructure analysis and a learning rate analysis. Assuminga hyperparameter analysis is required, training the SVMis likely much faster than training the neural network.On the other hand, the neural network is very robust,meaning a much smaller size of hyperparameter rangesmay be used.

VI. CONCLUSION

In conclusion, the results of this project have shownthat both neural networks and SVMs can be efficientlytrained to correctly label some 98% pulsars and non-pulsars in a data set consisting of statistical measuresradio signals’ IPPs and DM-SNR curves. Neither neuralnetworks not SVMs stand out as the better alternative.

Designing a neural network for this purpose, it hasbeen shown that using sigmoid activation, a small num-ber of layers (≈ 2) and few nodes ≈ 20 ± 5) is likely tooptimize the network for almost any choice of the initiallearning rate and learning momentum strength.

Designing an SVM for this purpose, it has been shownthat the best results were produced using a linear kernel.However, this finding is most likely due to the linearlyseparability of the specific pulsar data set analysed inthis project. Hence, if more flexibile SVMs are needed,it has been shown that similarly accurate results havebeen produced using a radial Gaussian with a violationof C = 1 and scaling factor γ = 0.1, and using a 3rd

degree polynomial kernel with a violation of C = 0.1 andpolynomial shift of r = 10.

[] F.Graham Smith, F.R.S., former Director of the RoyalGreenwich Observatory (1976-1981). Pulsars. CambridgeUniversity Press, 1977.

[] Jim Condon. Essential radio astronomy, chapter 6: Pul-sars. Online resource hosted by National Radio As-tronomy Observatory (US), Last modifed 2016. https:

//www.cv.nrao.edu/~sransom/web/Ch6.html.[] Integrated pulse profile of a pulsar. Original work

by Lorimer Kramer (Handbook of Pulsar Astronomy),republished by AstroBaki, Last Modified December6th 2017. https://casper.berkeley.edu/astrobaki/index.php/

Dispersion_measure.[] S.D. Bathes et. al. The high time resolution universe

survey vi: An artificial neural network and timing of 75pulsars. arXiv:1209.0793v2 [astro-ph.SR], 2012.

[] sckit-learn: Machine learning in python. Open Source.

18

Appendix A: Distributions of the Pulsar Data Set Predictors

Figure 11: Histograms of the IPP predictors in the pulsar data set. The pulsars have been separeated from thenon-pulsars in order to highlight the measurable difference between the two classes.

19

Figure 12: Histograms of the DM-SNR curve predictors in the pulsar data set. The pulsars have been separeatedfrom the non-pulsars in order to highlight the measurable difference between the two classes.