DevOps and Tools:Containers, .Net Core, NuGet, Xamarin, and

more…

Software Modeling and Analysis (CSE 681)Fall 2018

Overview

• DevOps Concepts and Ideas• Tools

• Container Technology • Concepts and Ideas

• Docker

• Relationship to DevcOps• Demonstration

• Survey of .Net Technologies• .Net framework (studied in CSE 681)• Net Core • Universal Windows Platform (UWP)• Xamarin (Mono)• .Net Standard

• Demo .Net Core with Docker• Use of Nuget

• Time Permitting• Demo Xamarin (shared code: Android, UWP)• Demo ASP.Net Core REST service

• Lists of References

DevOps? What’s is that…?

• Portmanteau for “Development and Operations” – DevOps

• a culture and set of processes with an objective • Optimize the efficiency of software development process within an organization

• develop, test, integrate, deploy, deliver software faster and more efficiently than through traditional approaches

• It’s popular movement…but not completely new idea!• … grew out of the Agile Manifesto [1]

• iterative/incremental approach focused on collaboration, customer feedback, and small, rapid releases.

• At organizational level!

• breaking down communication silos between teams

• Who’s in?

• Just about everybody: Amazon, Microsoft, Google, Facebook, Twitter Adobe, …., ….,

https://www.amazon.com/s/ref=nb_sb_noss_1?url=search-alias%3Daps&field-keywords=devopshttps://aws.amazon.com/devops/?nc2=type_ahttps://azure.microsoft.com/en-us/services/devops/?&OCID=AID736750_SEM_uI9fty9dhttps://devops.com/news/

DevOps Overview

• Main ideas Grew out of the disconnects that naturally arise between (people) • Miscommunication, misaligned goals and objectives manifest as conflict and

inefficiency. • compound to become increasing complex as the size of an organization increases (as

development teams increase in size)

• Evolved out of approaches to overcome these problems by focusing on the communication, collaboration, and integration between software development and information technology personnel [1]

• Goal is to highlight the break downs and bottlenecks (the Silos) that impede the development process [1]

1] - “Secure DevOps Process and Implementation” Software Engineering Institute (SEI), Carnegie Mellon University, January 11, 2017, https://www.sei.cmu.edu/training/P116.cfm

Breaking Down the silos!

• Silos inhibit collaboration and reinforce the waterfall methodology• A cultural movement in as much as it is a development cycle [1]

• “…that’s not my Job!” “That’s is my Job, stay out!...” (Ownership ideology)

• Everyone becomes a first class citizen and works collaboratively across the disparate areas: Requirements->Design->Implementation->verification>maintenance• e.g. Operations (security staff) should be involved in requirements and design phases else domain

expertise is lost!

[2] - http://www.agiletestingframework.com/devops-getting-old-deja-vu-feeling/ [3] - http://www.agilebuddha.com/agile/demystifying-devops

DevOps Principles Overview • DevOps (Development and Ops)– out growth of Agile manifesto [1]

• Agile embraces continuous change, DevOps embraces continuous test/integration/delivery/deployment• How? Break down the silos sub teams (Dev, Ops, Test, Q/A). Everyone becomes a first class

citizen.• Principles

➢ Collaboration• Use tools that influence collaboration (within and between teams), enforce

policy➢ Infrastructure as Code (IaC)

• entire environment (servers/links/platforms/apps/configuration) all represented as scripts stored in version control • Tools: Open stack (Heat), AWS (Cloud Formation), Docker, Chef, Puppet,

Vagrant➢ Automation

• Script as much manual processing as possible!• Continuous Integration/Test/Delivery/Deployment• Tools: Jenkins/Team City/AWS DevOps, Docker

➢Monitoring: • Cloud watch (AWS), Ceilometer (Openstack), VictorOPs

DevOps Core Principles [1]

• Collaboration • Efficiency (communication and processes) between disparate roles, both within teams between teams

in an organizations

• Infrastructure as Code (IaC)• Script and version control as much of infrastructure as possible, principally for capturing the disparate

environment details (.e.g. development, testing, production, etc.)

• Goal: (repeatable, recoverable, reusable, sharable)

• Automation• Automate human (error processes) wherever possible

• Goal: Increase frequency of testing, continuous (consistent/uniform) integration, delivery, and deployment processes

• Monitoring• Data to needed to inform development/operational decisions

• policies and priorities are domain dependent

• Is the System working? Performance? Are users consuming the system as expected

Source: https://www.sei.cmu.edu/education-outreach/courses/course.cfm?coursecode=V38

Parity: Automate (Standardize) Processes• Integration: merging disparate system artifacts is challenging

• Often done manually – (slow and error prone) == complex and expensive • Can’t automate everything, but … big changes have security implications that require review etc.)

• Continuous Integration (automate what makes sense…)• Automate/standardize submission process (get artifacts) into the various environments

(testing, staging…)• Greater visibility: everybody gets notified of issue(s) sooner: developers, testers (QA), Op

(Security), Management, etc. • fail fast and fix fast == increase efficiency.

• Continuous Delivery• Changes pushed often to customer/mock production environment

• perhaps staging is configured as such)• Give everybody (customer/stake holders) visibility

• Continuous Deployment• Changes are pushed continuously (often) into production

• Domain specific • doesn’t mean this makes sense • does mean the process for deployment is automated, consistent, reliable, and

repeatable.

• *Goal Process/Environment Parity to achieve frequent/regular testing • Make small, incremental changes

Tools that support/enable DevOps

• Not a single tool (cross-functional concerns)• multiple tools (tool chains) that fit into one or more categories reflective of the

development and delivery process…• Code – code development and review, source code management tools, code merging

• Build – continuous integration tools, build status

• Test – continuous testing tools that provide feedback on business risks

• Package – artifact repository, application pre-deployment staging

• Release – change management, release approvals, release automation

• Configure – infrastructure configuration and management, infrastructure as code tools

• Monitor – applications performance monitoring, end-user experience [3]

• https://en.wikipedia.org/wiki/DevOps

• https://www.quora.com/What-are-the-more-efficient-combinations-of-Devops-tools-to-use-in-a-web-project

DevOps Toolbox

https://hostadvice.com/blog/devops-toolbox-jenkins-ansible-chef-puppet-vagrant-saltstack/

https://devopedia.org/devops

Ultimate DevOps Tool list # 1

https://goo.gl/images/zH2Pys

Ultimate DevOps Tool list # 2

What is a Container? “Containerization”• Operating-system-level-virtualization – isolated (“sandboxed”)

process instances• share the OS kernel, no guest OS, no hyperviser

• Container Instances are built from distinct OS images

• Runs natively in Host OS, versus VM runs virtually in full blown guest (via hypervisor)

• Uses kernel namespaces ( for isolation) and cgroups (control groups) for resource management.

• Images consists only of the application and the library dependencies required to run the app

• Lightweight (efficient), flexible, portable execution environment

• Decreased code footprint lends to enhanced security

• Foundational component of microservices architectures

• Not a new idea…remember (BSD jail, Unix chroot), LXC (Linux Containers) • https://en.wikipedia.org/wiki/Operating-system-level_virtualization

https://goo.gl/images/FCgPFY

Images and Layers

• Containers are built from layered images (think .exe image versus process)• Each layers is an application dependency (read only)

• Union filesystem (Overlay FS – OverlayFS, AUFS, BtrFS storage driver)• CoW (copy-on-write)

https://docs.docker.com/v17.09/engine/userguide/storagedriver/imagesandcontainers/#images-and-layers

Layer Image (Concept)

What is Docker?• An open platform developing, shipping and running applications using

container technology• ship, testing, and deploy code quickly

• reduce the time between code development and running in production.

Docker Engine is a client-server application with these major components:

1. A server which is a type of long-running

program called a daemon process (the dockerd command)

2. A REST API which specifies interfaces that

programs can use to talk to the daemon and instruct it what to do.

3. A command line interface (CLI) client (the docker command).

4. Registry (Docker Hub)

https://docs.docker.com/engine/docker-overview/

Container is not a Virtual machine!

https://docs.docker.com/get-started/#images-and-containers

Benefits and Risks

• Benefits • Flexible: Even the most complex applications can be containerized.

• Lightweight: Containers leverage and share the host kernel.

• Interchangeable: You can deploy updates and upgrades on-the-fly.

• Portable: You can build locally, deploy to the cloud, and run anywhere.

• Scalable: You can increase and automatically distribute container replicas.

• Stackable: You can stack services vertically and on-the-fly.

• Risks -> Security is the big one• Subject of on going discussion..

• Docker Engine (daemon itself), registry (.e.g. docker hub), intrinsic security of kernel

• Limited ability to reason effectively across abstraction boundaries

https://docs.docker.com/get-started/https://docs.docker.com/engine/security/security/

Example Scenario…

✓Developers write code locally and shares their work with their colleagues using Docker containers.

✓Use Docker to push their applications into a test environment and execute automated and manual tests.

✓When developers find bugs, they can fix them in the development environment and redeploy them to the test environment for testing and validation.

✓When testing is complete, getting the fix to the customer is as simple as pushing the updated image to the production environment.

https://docs.docker.com/engine/docker-overview/#docker-engine

Flexible OS Stack (Run Linux or Windows)

XenServer Hypervisor

Linux Virtual Machine

Docker Engine

Base Image: Bins/Libs /Dependencies

QEMU (Bin)

Windows OS Guest

Image Layers

QEMU: Machine Hardware Configuration 1

Linux OS Guest

QEMU: Machine Hardware Configuration 2

• Update respective layers to extend new machine hardware and Operating System type

• Running container instance: build from image• Shares host OS with

other containers

XContainer 1

XX

eth0 tap0

VM_NET (eth0)

X XX

eth0 tap0

br0

QEMU QEMU

Hardware Emulation (x86 Intel) X

Container 2

X

Virtual Machine (VM) Host

X

X

X

br0

tap0

br0X

tap1

E1000 NIC NIC

XNIC (eth0)

qemu-system-x86_64 -m 8192 -vnc :2 –net nic,vlan=0,macaddr=00:80:7F:31:41:72,model=e1000 -net

tap,vlan=0,ifname=tap0,script=/etc/qemu-ifup -usbdevice tablet [windows-image.qcow2] | [android-

image.qcow2]

Hardware Emulation (x86 Intel)

Hypervisor

VM_NET (eth0)

Virtual Machine (VM) Node: Mint 17.1

X

Xtap0

XNIC (eth0)

X X X X X X X XXbridge0

Container 1

Container 9

Container 5Container 4Container 3Container 2

Container 6 Container 7 Container 8

Physical interface

Base image: Ubuntu 14.04

Layer 1: Qemu

Layer 2: Windows 7

Base image: Ubuntu 14.04

Layer 1: Firebox Windows

image Firefox image

Ubuntu 14.04base image

Demo Docker Containers

What didn’t we discuss about Docker?

• Orchestration tools and Microservices1. Docker machine

• install Docker Engine on virtual hosts, and manage the hosts with docker-machine commands

• local machines (Mac or Windows)

• company network (data center)

• Public cloud providers like Azure, AWS

https://docs.docker.com/machine/overview/

2. Docker Compose

• tool for defining and running multi-container Docker applications• Define YAML formatted file to configure

your application’s services. • Execute the file as (single command) to

create and start all the services from your configuration• This is microservices!

• Transient, lightweight services composed of cooperative containers!

• This is DevOps• Automation (repeatable and

reproducible)• Infrastructure as Code (IaC)

• Environment Parity

https://docs.docker.com/compose/overview/

3. Docker Swarm

• Cluster management integrated with Docker Engine• Decentralized design• Declarative service model• Scaling• Multi-host networking• Service discovery• Load balancing• Security

• TLS mutual authentication and encryption to secure communications

• Rolling updates• At rollout time you can apply service updates to nodes incrementally

https://docs.docker.com/engine/swarm/#feature-highlight

Cloud Orchestration

• Open stack (leading open source on-premise (private cloud) -- IaaS• Template based (YMAL format)

• describes the infrastructure for a cloud application in a text file that is readable and writable by humans, and can be checked into version control, diffed

• Infrastructure resources that can be described include: servers, floating ips, volumes, security groups, users, etc.

• primarily manages infrastructure, but the templates integrate well with software configuration management tools such as Puppet and Chef.

• Seeks for compatibility with AWS (Cloud formation)

• i.e. Portable cloud infrastructure provisioning

• Easy to support containers

• Built in container service: Openstack Magnum

• https://github.com/openstack/heat-templates/blob/master/hot/F18/WordPress_Native.yaml

• https://wiki.openstack.org/wiki/Heat

• https://wiki.openstack.org/wiki/Magnum

Mike Metral https://getcarina.com/docs/best-practices/container-technologies-orchestration-clusters/

Microservices (Orchestration)

Main goal: provide automated container management as well as guarantees for multi-container services and container engines.

• Marathon with Apache Mesosphere(Scheduler) and DC/OS

• Multiple physical node scaling (treated as one machine)

• Kubernetes (Google Inc.)

• Large scale service oriented design

• “Self-Healing” services and Load balancing

• Swarm (Docker)

• Standard basic clustering tool

• Native Docker API for third party tool integration

https://docs.mesosphere.com/overview/, https://github.com/coreos/fleet/blob/master/Documentation/fleet-k8s-compared.md

Google Docs (An Example)

Shared Hardware

Persistent StorageKubernetes Node

Kubernetes Agent

Kubernetes Node

Kubernetes Agent

Kubernetes Master/Proxy

Bob Alice

= Container = Service Pod

A survey of .Net Implementations • Microsoft actively develops and maintains four (4) distinct .Net implementations

1. .Net framework• Current version 4.7.x

• Windows (desktop) applications. Includes Windows specific APIs, such those the utilized with Winforms, WPF, and WCF.

2. .Net Core • Cross platform (Windows, Linux, macOS)

• Current version 2.1

• server and Cloud (microservices), service oriented (Web) applications ASP.NET Core

• Version 3 will include support for Winforms and WPF (Windows only)

3. Universal Windows Platform (UWP) • building modern, touch-enabled Windows applications and software for the Internet of Things

(IoT). unify the different types of devices that you may want to target, including PCs, tablets, phablets, phones, and even the Xbox.

• provides many services, such as a centralized app store, an execution environment (AppContainer),

• C# and VB.NET, the .NET APIs are provided by .NET Core

Survey of .Net Implementations (cont.)

Xamarin (Mono)• Often use cases where a small runtime is required

• Used for Xamarin applications on Android, Mac, iOS, tvOS and watchOS and is focused primarily on a small footprint.

• also powers games built using the Unity engine.

• Xamarin Native and Xamarin Forms:

https://docs.microsoft.com/en-us/xamarin/#pivot=platforms&panel=Android

• .Net standard (replacement for portable class libraries (PCL) ) – latest version: 2.0

• An API (BCL) standard implemented by all .Net implementations

• enables portability across different .NET implementations, effectively allowing code to run everywhere.

.Net standard Source: https://docs.microsoft.com/en-us/dotnet/standard/components

Advice Source: https://docs.microsoft.com/en-us/dotnet/standard/choosing-core-framework-server.Net Native: https://docs.microsoft.com/en-us/dotnet/framework/net-native/

.Net Implementations

https://social.msdn.microsoft.com/Forums/vstudio/en-US/7035edc6-97fc-49ee-8eee-2fa4d040a63b/what-are-differences-between-net-framwork-net-standard-and-net-core?forum=clr

.Net Implementations -- Architecture Overview

https://blogs.msdn.microsoft.com/cesardelatorre/2016/06/27/net-core-1-0-net-framework-xamarin-the-whatand-when-to-use-it/

Xamarin Native Versus Xamarin Forms

Xamarin Architecture

https://docs.microsoft.com/en-us/xamarin/android/internals/architecture

Prototype: UWP, Android, iOS

What is .Net Core?

• open-source, general-purpose .Net platform maintained by Microsoft and the .NET community on GitHub• Cross-platform: Windows, macOS and Linux• Consistent across architectures: multiple architectures, including x64, x86,

and ARM.• Command-line tools: command-line tools for local development and in

continuous-integration scenarios.• Flexible deployment: install machine wide or install side-by-side

• include with app, Designed for containerization

• Compatible: .NET Core is compatible with .NET Framework, Xamarin and Mono, via .NET Standard

• Open source: MIT and Apache 2 licenses.

https://docs.microsoft.com/en-us/dotnet/core/about

Core CLI (command line interface)

• Cross platform toolchain for developing .Net applications• Principle interaction component for

• end users, higher level tools (IDE such as Visual Studio), other orchestration environments ( e.g. Docker )

• a NuGet client (stay tuned)

• CLI Command structure: the driver (“dotnet”), following by a command (“verb”)• Driver performs two functions: (1) run a command, (2) run a framework dependent app

• “dotnet new console” -- create a new console app

• “dotnet build --output /build_output“ -- build output

• “dotnet /build_output/my_app.dll“ -- run the app

https://docs.microsoft.com/en-us/dotnet/core/tools/?tabs=netcore2x

NuGet

• Package Manager for .Net• enables developers to create, share, and consume code

• Microsoft supported mechanism for sharing .Net code as “packages”

• A "packages" contains compiled code (as DLLs) along with other content needed in the projects that consume these packages

• defines how packages for .NET are created, hosted, and consumed

• A single zip file with a .nupkg extension• Contains a compiled code (.dll), Related files, and descriptive manifest

• Dependency Management• Builds and maintains dependency graph on behalf of the project

• Distributed with Visual Studio• Can host packages privately in the cloud (such as on Azure DevOps),

https://docs.microsoft.com/en-us/nuget/what-is-nuget

NuGet Model

https://docs.microsoft.com/en-us/nuget/what-is-nugetSource:

The Basic process..• “dotnet new console”

• Creates an up-to-date project: “Hello.csproj”• The project files specifies everything need to restore dependencies, and build

the program• Uses Nuget tools: MSBuild: (Creation/Consumption – restore ) as part of CLI driver

“dotnet”• OutputType tag specifies the project type: library or executable (console)• TargetFramework tag specifies the .Net implementation being targeted.

• As of .net Core 2.0. each of the CLI driver command: “new”, “build”, “run” implicitly run “restore” (i.e. dotnet restore) which performs the following operations using NuGet• analyzes the .csproj file• downloads the file dependencies (or grabs from your machine cache)• writes the obj /project.assets.json file

• the package reference lists

https://docs.microsoft.com/en-us/dotnet/core/docker/docker-basics-dotnet-corehttps://docs.microsoft.com/en-us/nuget/reference/msbuild-targets

Visual Studio Tools for Docker

https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/docker/visual-studio-tools-for-docker?view=aspnetcore-2.1

• create .Net core project• Add -> container orchestration support

Demonstrate .Net Core with Docker

Sources

[1] - “Secure DevOps Process and Implementation” Software Engineering Institute (SEI), Carnegie Mellon University, January 11, 2017, https://www.sei.cmu.edu/training/P116.cfm

[2] - http://www.agiletestingframework.com/devops-getting-old-deja-vu-feeling/

[3] - http://www.agilebuddha.com/agile/demystifying-devops/

[4] - https://github.com/openstack/heat-templates/blob/master/hot/F20/WordPress_Native.yaml

[5] - https://docs.mesosphere.com/overview/

[6] - https://github.com/coreos/fleet/blob/master/Documentation/fleet-k8s-compared.md

[7] - https://en.wikipedia.org/wiki/OpenStack