S3IT: Service and Support for Science IT

OpenStack for academicuse-cases ?Sergio MaffiolettiS3IT: Service and Support for ScienceIT,University of Zurich

22.03.2016

What is S3IT and who am I

S3IT - Service and Support for ScienceIT:http://www.s3it.uzh.ch.

Enabling research through Science IT competences.

S3IT is an academic solution provider (end-to-end).

Sergio MaffiolettiSenior Infrastructure and Application specialist atUniversity of Zurich.

UZH/S3IT OpenStack for academic use-cases 22.03.2016

Overall message

Build and adaptinfrastructure to use-cases

and not the other way round.

UZH/S3IT OpenStack for academic use-cases 22.03.2016

Let’s start with someexamples

UZH/S3IT OpenStack for academic use-cases 22.03.2016

Research needs to do more than just dataanalysis.

Personalized server

Large-scale data analysis and simulations

Integrate solution

UZH/S3IT OpenStack for academic use-cases 22.03.2016

Research needs to do more than just dataanalysis.

Personalized server→ RStudio/R Shiny→ Windows image analysis

Large-scale data analysis and simulations

Integrate solution

UZH/S3IT OpenStack for academic use-cases 22.03.2016

Research needs to do more than just dataanalysis.

Personalized server

Large-scale data analysis and simulations

→ Hadoop/Spark→ Own SLURM cluster→ Matlab and toolboxes

Integrate solution

UZH/S3IT OpenStack for academic use-cases 22.03.2016

Research needs to do more than just dataanalysis.

Personalized server

Large-scale data analysis and simulations

Integrate solution

→ from instrument to store, to processing to makedata accessible.

UZH/S3IT OpenStack for academic use-cases 22.03.2016

What is OpenStack ?

UZH/S3IT OpenStack for academic use-cases 22.03.2016

What is OpenStack and how we use it

→ Self-provisioning of Virtual Instances, Storage andNetwork→ Elastic allocation of resources on demand→ Multi-tenancy

UZH/S3IT OpenStack for academic use-cases 22.03.2016

OpenStack project

OpenSource projectask our sysadmin why this is important. . .

Foundation supported by more than 200 companies.https://www.openstack.org/foundation/companies/

Network APIs→ Amazon compute and storage cloud compliant→ Bindings for many languages

UZH/S3IT OpenStack for academic use-cases 22.03.2016

Let’s implement some of theexamples

UZH/S3IT OpenStack for academic use-cases 22.03.2016

What approach ?

→ Provision, Configure and Orchestrate basecomponents: network, storage, compute.→ Build on top of baseline infrastructure.→ Customize and tailor to use-cases needs.

UZH/S3IT OpenStack for academic use-cases 22.03.2016

Personalized server: R Shiny

UZH/S3IT OpenStack for academic use-cases 22.03.2016

Personalized server: R Shiny

1. Create own instance

2. Automated customization (e.g. ansible)

3. Application-specific settings

4. Save base image for further use

UZH/S3IT OpenStack for academic use-cases 22.03.2016

R Shiny: Create own instance

Instances are virtual machines that run inside the cloud.

UZH/S3IT OpenStack for academic use-cases 22.03.2016

R Shiny: Automated customization (e.g.ansible)

Ansible is a free-software configuration managementplatform for configuring and managing computers.

UZH/S3IT OpenStack for academic use-cases 22.03.2016

R Shiny: Application-specific settings

Security groups are sets of IP filter rules that are applied toan instance’s networking.

UZH/S3IT OpenStack for academic use-cases 22.03.2016

R Shiny: Save base image for further use

The OpenStack snapshot mechanism allows to create newimages from running instance.

UZH/S3IT OpenStack for academic use-cases 22.03.2016

Create your own batch cluster: SLURM

UZH/S3IT OpenStack for academic use-cases 22.03.2016

Create your own batch cluster: SLURM

1. Provision cluster instances (1 frontend + Xcompute)

2. Create own data volume and attach it to frontend

3. Install and configure frontend and compute

4. Export via NFS data volume to all compute nodes

5. Start NFS and slurm services

UZH/S3IT OpenStack for academic use-cases 22.03.2016

Create your own batch cluster: SLURM

. . . or use elasticluster: http://gc3-uzh-ch.github.io/elasticluster/

https://youtu.be/gkXkcHDd588

UZH/S3IT OpenStack for academic use-cases 22.03.2016

Why flexibility: from our perspective

→ Reduce total time to get actual results→ Allow to run on multiple infrastructures(portability)→ Automate (more research, less development forend-user)

UZH/S3IT OpenStack for academic use-cases 22.03.2016

Why flexibility: from user perspective

Self-provisioning and Elasticity of resources

Customization and control of the environment

Network API

UZH/S3IT OpenStack for academic use-cases 22.03.2016

Why flexibility: from user perspective

Self-provisioning and Elasticity of resources→ End-users can allocate and release resourceswhen needed.

Customization and control of the environment

Network API

UZH/S3IT OpenStack for academic use-cases 22.03.2016

Why flexibility: from user perspective

Self-provisioning and Elasticity of resources

Customization and control of the environment→ End-users can tailor the research infrastructureto their specific needs.

Network API

UZH/S3IT OpenStack for academic use-cases 22.03.2016

Why flexibility: from user perspective

Self-provisioning and Elasticity of resources

Customization and control of the environment

Network API→ To programmatically create and control an ownresearch infrastructure.→ Build services on top

UZH/S3IT OpenStack for academic use-cases 22.03.2016

Downsides

→ Resource allocation: policy is basically done bysupport team.→ Efficient use of resources: users need to takeresponsibility.→ Need to have strong support team to complementthe offer to end-users.→ Ops costs comparable as for regular batch clusterbut know-how has to be developed.→ Storage solution has to be planned separately andmostly depends on use-cases.

UZH/S3IT OpenStack for academic use-cases 22.03.2016

Conclusions

→ Cloud is flexible: build and adapt infrastructure touse-cases and not the other way round.→ Users have more possibilities but also moreresponsibilities.→ Easier for end-users to think and plan theirresearch infrastructure.

UZH/S3IT OpenStack for academic use-cases 22.03.2016