Soft

ware

Engin

eeri

ng @

FO

M Inst

itute

AM

OLF

Marco Konijnenburg

SIG SE @ Nikhef

14 november 2013

KITE

A down to earth approach to cloud computing

Soft

ware

Engin

eeri

ng @

FO

M Inst

itute

AM

OLF

2

Contents

• Context

• Predecessor

• Goals

• Design

Soft

ware

Engin

eeri

ng @

FO

M Inst

itute

AM

OLF

3

Processing large datasets

• Datasets are over 10 GB in size

• Collection of thousands of spectra

• Each spectrum can be processed individually

Soft

ware

Engin

eeri

ng @

FO

M Inst

itute

AM

OLF

4

Compute

• Use general purpose compute clusters

• Use dedicated compute cluster

• Splittable compute jobs

Soft

ware

Engin

eeri

ng @

FO

M Inst

itute

AM

OLF

5

Cloud

• HPC –Cloud Calligo @ SARA – Open Nebula implementation

• Configurable systems– Mix various systems (like different OS-es)

• Start and stop systems at will– Using VM’s

Soft

ware

Engin

eeri

ng @

FO

M Inst

itute

AM

OLF

6

COMMIT

Job workersJob office (headnode)

Calligo

Firewall

Proxy

Internet

ToPoS

J ob administratorJob administrator

Soft

ware

Engin

eeri

ng @

FO

M Inst

itute

AM

OLF

7

Job submission application• Connect to headnode

• Specify script , data and storage for MS processing

• Specify number of tokens (job parts)

• Submit job– Receive email of job

status– View progress

Soft

ware

Engin

eeri

ng @

FO

M Inst

itute

AM

OLF

8

Cloud UI

Soft

ware

Engin

eeri

ng @

FO

M Inst

itute

AM

OLF

9

Improve

• Easier interface for non computer scientists– Start and stop preconfigured group of VM’s

• Interface from desktop app to web app– Multi platform– Always up to date– Needs a login to remember user settings

Soft

ware

Engin

eeri

ng @

FO

M Inst

itute

AM

OLF

10

Overview

Soft

ware

Engin

eeri

ng @

FO

M Inst

itute

AM

OLF

11

Kite log in

• Use existing techniques

– Browser login, SSO

– Using Shibboleth to connect to SURFConext

Soft

ware

Engin

eeri

ng @

FO

M Inst

itute

AM

OLF

12

Soft

ware

Engin

eeri

ng @

FO

M Inst

itute

AM

OLF

13

KRON

• Kite relay for Open Nebula

• Only command and data passing

• Reports to Kite when (re)started

Soft

ware

Engin

eeri

ng @

FO

M Inst

itute

AM

OLF

14

KRON

Soft

ware

Engin

eeri

ng @

FO

M Inst

itute

AM

OLF

15

KRON (expanded)

Soft

ware

Engin

eeri

ng @

FO

M Inst

itute

AM

OLF

16

Communication

Soft

ware

Engin

eeri

ng @

FO

M Inst

itute

AM

OLF

17

Issues & Wishes• XML RPC requires extra log in, not

SURFConext’ed

• Within one group VM’s ‘see’ each other

• Data in and out

• Scale it up to multiple clouds, possibly multiple types

• Scale it up to users from other institutes

Soft

ware

Engin

eeri

ng @

FO

M Inst

itute

AM

OLF

18

The end

Thank you for your attention

Contact:

M.KONIJNENBURG@AMOLF.NL

MARCO.KONIJNENBURG

HTTP://WWW.LINKEDIN.COM/IN/MARCOKONIJNENBURG