Data Protection for JFrog Artifactory on ONTAP 9 with ... · Data Protection for JFrog Artifactory...

Technical Report

Data Protection for JFrog Artifactory on ONTAP 9 with SnapCenter 2.0 using RESTful APIs Application-Consistent Backup of MySQL Databases and Artifactory Repositories

Bikash Roy Choudhury, NetApp

Amarnath Rachapudi, NetApp

Jainish Shah, JFrog

July 2017 | TR-4607

Abstract

JFrog Artifactory is the most popular universal binary repository manager, supporting different software

packages created in any language. Developers write and test code using artifacts—libraries, compilers, and

tools—and deliver a successful final build. JFrog Artifactory is commonly used in DevOps processes as a

consolidated build repository. One of its key features is consistent version control for all the build packages

that development teams work on prior to production release.

Together with JFrog Artifactory, NetApp software manages all the build data that is created. In addition to

providing a robust and scalable data management plaftform, NetApp® ONTAP® helps developers meet

requirements such as data protection and data availability. By configuring JFrog Artifactory data volumes on

NetApp systems, developers can use NetApp SnapCenter® 2.0 to protect from data loss by taking

application-consistent snapshots of the database and the repository.

This report discusses the integration of JFrog Artifactory with MySQL, where the database and the checksum

repository is on ONTAP 9 a robust continuous integration/continuous deployment (CI/CD) pipeline.

SnapCenter 2.0 is used as a Microsoft Windows-based tool for taking application-consistent snapshots of the

database and the repository volumes in ONTAP.

In partnership with

https://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0ahUKEwj28PSR1u3SAhVO0GMKHflaADsQjRwIBw&url=https://twitter.com/jfrog&psig=AFQjCNHdg2eKHDQqadfCCl5-01WI4kgoJw&ust=1490394450448031

2 JFrog Artifactory on ONTAP 9 Using SnapCenter © 2017 NetApp, Inc. All rights reserved. © 2016 NetApp, Inc. All rights reserved.

TABLE OF CONTENTS

1 Introduction ........................................................................................................................................... 3

2 ONTAP 9 and JFrog Artifactory .......................................................................................................... 4

3 SnapCenter 2.0 and Artifactory 5.1.4 .................................................................................................. 5

4 Summary ............................................................................................................................................... 9

5 Conclusion .......................................................................................................................................... 10

References ................................................................................................................................................. 11

Appendix .................................................................................................................................................... 11

LIST OF FIGURES

Figure 1) JFrog Artifactory in the continuous integration/continuous deployment (CI/CD) workflow. .............................3

Figure 2) Continuous integration workflow with JFrog Artifactory. ..................................................................................4

Figure 3) SnapCenter 2.0 integration with JFrog Artifactory. ..........................................................................................6

Figure 4) SnapCenter 2.0 and the MySQL plug-in. ........................................................................................................7

Figure 5) List of Artifactory volumes configured in SnapCenter 2.0. ...............................................................................7

Figure 6) A consistency group is enabled from SnapCenter 2.0 for Artifactory volumes. ...............................................8

Figure 7) SnapCenter 2.0 creates a snapshot with job ID 601. ......................................................................................9

Figure 8) Snapshot listing of the Artifactory volumes in NetApp OnCommand® System Manager. ...............................9


1 Introduction

In a typical DevOps software development lifecycle process, the development languages used by development teams can require different artifacts in the form of repositories, tools, and libraries to compile and package the application dependencies. These dependencies come into play while the team is promoting the product to quality assurance/user acceptance testing and then releasing it into production. Managing different versions of binaries from different repositories created by various development teams is a challenge. JFrog Artifactory is a universal binary repository manager that supports different software packages created through different development tools, languages, and technologies.

As illustrated in Figure 1, Artifactory plays the role of a central hub for a binary artifact repository. It

manages builds from the CI pipeline and then promotes them to the CD process with well-defined security

policies for authorized personnel. The binary artifact repository is very different from the source code

repository: The source code repository manages version control for source code only, and does not

include prebuild artifacts such as run-time libraries, tools, and compilers.

During the CI process, several types of data can be generated each time a build is updated or revised.

The combination of binaries (final builds); the tools, libraries, and compilers used to develop the code;

and information about the software and systems used at the time an application was created are

collectively referred to as an artifact. The complexity of managing these artifacts increases with the size of

development teams, the increased frequency of builds, and the amount of 3rd-party dependencies (i.e.

programming languages, libraries, frameworks, etc).

This technical report focuses on a use case involving the implementation of JFrog’s Artifactory with

NetApp® ONTAP® 9 data management software. ONTAP provides data management, data protection,

and high data availability, and enables performance at scale. Also included in the implementation is

NetApp SnapCenter® 2.0. Together with ONTAP, SnapCenter 2.0 and storage-level RESTful APIs

provide monitoring and reporting, provisioning with service-level objectives, and data protection.

In the context of storage, JFrog Artifactory can be thought of as consisting of two primary components: a

database that stores the metadata pertaining to artifacts and builds, and the repository that stores the

files as checksums. JFrog uses Apache Derby DB as the default database, but it can connect with any

commercial SQL database for the metadata store.

Developer Workspace(Private Builds)

QA/Staging Deploy

Source Code Repository

Binary Artifact Repository

Continuous Integration (Integrate Builds)

1

24

5 6

1

2Pull artifacts, integrate Builds;

Push Builds to JFrog Artifactory

3 User Workspace for Private Builds

4 Commit Code Changes3

Figure 1) JFrog Artifactory in the continuous integration/continuous deployment (CI/CD) workflow.


This report discusses the integration of Artifactory with MySQL, where the database and the checksum

repository are on ONTAP 9 for the CI/CD pipeline. SnapCenter 2.0 is used as a Windows-based tool for

taking application-consistent snapshots of the database and the repository volumes in ONTAP.

2 ONTAP 9 and JFrog Artifactory

NetApp ONTAP 9 is data management software that is optimized for many types of workloads, such as

databases, file shares, analytics, and application development. ONTAP supports these workloads in both

physical and virtual environments that can be on the premises or in a hybrid cloud. Along with data

reliability, efficiency, and availability, ONTAP provides horizontal scaling and performance required for all

supported workloads on a standard run-time system.

Because workloads are increasingly running on containers, NetApp now provides NetApp Docker Volume

Plugin to enable persistent data volumes to be mounted to Docker containers for databases, source code,

and binary artifact repositories. ONTAP RESTful APIs provide the flexibility to provision, load-balance,

and scale workloads in a single cluster or across clusters in multisite environments.

As illustrated in Figure 2, Docker Swarm manages resiliency for Docker containers by creating a multihost

cluster across virtual or physical machines. In the CI process, the sequence is as follows:

1. JFrog Artifactory is configured in Docker containers by using the docker-compose.yml

installation file. The installation provides private repositories for all the artifacts that are required

during the software build process and application packaging.

2. The data volumes for the Docker containers are then created by NetApp Docker Volume Plugin

before Artifactory installs the database and the repository.

3. GIT and Jenkins tools are set up and configured in Docker containers automatically. This occurs

when the respective data volumes are provisioned for source code version control, integrated

builds, and private builds through ONTAP APIs and mounted on Docker containers through

NetApp Docker Volume Plugin.

NetApp DockerVolume Plugin

CodeRepo1

CodeRepo2

CI Volume

for

Code Repo1

CI Volume

for

Code Repo2

Continuous Integrations (CI) Developer Workspaces

Integrated BuildsSource Code

Repository Manager Private Builds

Pre-Packaged

Source Code

& Pre-Build

ArtifactsArtifacts

& Builds

Universal BinaryRepository Manager

Snapshots Snapshots Snapshots

DatabasePre-Packaged

Source Code

& Pre-Build

Artifacts

Figure 2) Continuous integration workflow with JFrog Artifactory.


The Artifactory plug-in for Jenkins allows a developer to pull prebuilt artifacts created during code

compilation and push an updated software build in the Artifactory repository.

The entire CI process with GIT, Jenkins, and JFrog Artifactory on ONTAP using APIs and Docker is

automated by NetApp.

Application-consistent checkpoints, or snapshots, for JFrog Artifactory are created for the following

reasons:

• Data can be backed up to a secondary storage tier like tape or replicated to a disaster recovery

site for protection.

• Data can be recovered and restored quickly if the database or the checksums in the Artifactory

volume become corrupted. if there is data corruption with the database or the checksums in the

Artifactory volume.

• An instant copy of the database can be spun up for testing patches or database refactoring

before new changes are pushed to production.

Developers can perform all of these data manageability functions by using SnapCenter 2.0 from a single

management pane. NetApp and JFrog have jointly worked to provide a seamless experience, allowing

developers to create application-consistent checkpoints from Artifactory whenever a change occurs in an

Artifactory data volume. Developers can create these checkpoints without the aid of IT administrators or

DevOps engineers.

3 SnapCenter 2.0 and Artifactory 5.1.4

NetApp SnapCenter software is a unified, scalable system for application-consistent data protection and

clone management. SnapCenter simplifies backup, restore, and clone lifecycle management with

application-integrated workflows. Using storage-based data management, SnapCenter increases

performance and availability and reduces testing and development times.

MySQL is a popular database choice because the LAMP (Linux, Apache, MySQL, PHP/Perl/Python) stack is one of the most popular open-source web-based platforms. MySQL is commonly used in mobile and web-based application development and deployment. Artifactory can connect to existing databases such as Oracle or SQL Server. In this report, a MySQL database is used for Artifactory. The code in this report creates artdb as one of

the databases in MySQL. SnapCenter 2.0 supports custom plug-ins for databases such as MySQL and DB2. More plug-ins will be available for NoSQL databases in the future.

SnapCenter 2.0 is installed on a Windows node, whereas JFrog Artifactory 5.1.4 components are

installed and configured on a Linux node. As shown in Figure 3 The Linux node is also configured with

the Docker engine and NetApp Docker Volume Plugin. The database component of Artifactory could be

Oracle, MySQL, or SQL for the artdb database store and can manage metadata of all the artifacts and

builds.

When docker-compose.yml is executed to install Artifactory, it not only installs Artifactory in the

Docker container, but it also creates persistent data volumes on ONTAP 9 and mounts them on the

containers by using NetApp Docker Volume Plugin. The installation process also creates artdb in

MySQL as a new database. The contents of the docker-compose.yml file are listed in the Appendix.

http://mysupport.netapp.com/tools/info/ECMLP2633631I.html?productID=62127


Figure 3) SnapCenter 2.0 integration with JFrog Artifactory.

The MySQL database is set up on a physical machine for this validation. It can also run on a virtual

machine (VM). The following code shows Artifactory connecting with the MySQL database and lists the

artdb database after the installation.

BuildArtifacts

Snapshots

Database

Backup to Tape

Repository

Consistency Group

Snapshots

MySQL Agent

MySQL

SnapCenter2.0

NetApp DockerVolume Plugin

Dev/Test

Instantaneous Copies of DB Disaster Recovery

to Cloud

[root@linux-node6 ~]# mysql -u artifactory -p

Enter password:

Welcome to the MySQL monitor. Commands end with ; or \g.

Your MySQL connection id is 1944

Server version: 5.7.17 MySQL Community Server (GPL)

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights

reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input

statement.

mysql> show databases;

+--------------------+

| Database |

+--------------------+

| information_schema |

| artdb |

+--------------------+

2 rows in set (0.01 sec)


To take an application-consistent snapshot, SnapCenter must communicate with the database.

SnapCenter identifies the MySQL plug-in during the host discovery phase. Figure 4 shows SnapCenter

remotely installing the agent for MySQL in the database location. When the agent is in a “running” state,

SnapCenter 2.0 is ready to create snapshots.

As part of the resource configuration process, the DevOps admin creates a MySQL database resource

that contains MySQL artdb details, the storage, and the Artifactory volumes.

Figure 4) SnapCenter 2.0 and the MySQL plug-in.

Figure 5) List of Artifactory volumes configured in SnapCenter 2.0.


Next, a MySQL policy is created to specify the backups and the schedule. To provide the backup

replication details in the policy, a resource group with the MySQL resource is created and associate the

MySQL policy to that group.

Next, a MySQL policy is created to specify the backups and the schedule. The backup replication details

also have to be provided in the policy. Create a resource group with the MySQL resource and associate

the MySQL policy to that group. If the group complies with the business requirements, the resource

group is configured appropriately

The MySQL and Artifactory repository volumes are configured as a consistency group in SnapCenter 2.0,

as shown in Figure 6. With the help of the MySQL agent, SnapCenter 2.0 flushes all the logs and memory

buffers into the disk to take an application-consistent snapshot.

In Figure 6, a consistency group is enabled for both the Artifactory data volumes that were listed in Figure

5: mysql_vol and root_artdata. When a snapshot is triggered by SnapCenter 2.0, two snapshots are

created instantly for both the Artifactory volumes.

To prevent data loss, Artifactory does not update the metadata information in artdb until new or modified

checksums are written to the binary repository. Whenever SnapCenter 2.0 takes a snapshot, the

The artifactory.json file consists of the SnapCenter 2.0 API calls for a nonexpiring authorization

token followed by resource group and resource policy settings. For details about artifactory.json,

see the Appendix.

Here is how to create an application consistent snapshot from the Artifactory command line interface

(CLI).

Figure 6) A consistency group is enabled from SnapCenter 2.0 for Artifactory volumes.


After the developer executes the curl command from the CLI of the Artifactory node, job ID 601 is

initiated. Figure 7 shows that SnapCenter 2.0 received the request to create an application-consistent

snapshot with job ID 601, and created snapshots for both the Artifactory data volumes.

Figure 7

Figure 8 lists the snapshots of the Artifactory data volumes on ONTAP 9 where the data volumes

physically reside.

4 Summary

[root@linux-node6 ~]# curl -u admin:password -H "Content-Type:

application/json" -X POST -d @artifactory.json

http://10.192.39.26:8081/artifactory/api/plugins/execute/snapCenter

[Backup:null, Context:null, DisplayCount:0, Job:[ApisJobKey:null,

Description:null, EndTime:null, Error:null, EventId:0, Id:601,

IsCancellable:false, IsCompleted:false, IsRestartable:false, IsScheduled:false,

IsVisible:true, JobTypeId:null, Name:Backup of Resource Group 'ART_RG' with

policy 'ART_Policy', ObjectId:0, Owner:null, ParentJobID:0,

PercentageCompleted:0, PluginCode:0, PluginName:null, Priority:0,

StartTime:2017-05-04T10:24:29.3567493-07:00, Status:5, Tasks:[]],

Result:[ErrorRecords:[], _errorCode:0, _message:], Results:[], TotalCount:0]

Figure 7) SnapCenter 2.0 creates a snapshot with job ID 601.

Figure 8) Snapshot listing of the Artifactory volumes in NetApp OnCommand® System Manager.


The NetApp and JFrog joint integration with Artifactory and SnapCenter 2.0 allows developers to take

application-consistent snapshots on demand whenever they push modified or new artifacts and builds to

the repository. The developer has more control over the data (including the artifacts themselves), which is

available in a consistent state during the application development and deployment process. The

Artifactory plug-in gives developers more autonomy from the operations team, thus reducing time-

dependent communication challenges.

There is a one-time setup and configuration for SnapCenter 2.0 and Artifactory data volumes. After

SnapCenter 2.0 is implemented with the right resource group and resource policy settings, the developer

can use the Artifactory plug-in to make RESTful API calls to SnapCenter to take application-consistent

snapshots. The Artifactory plug-in is available in GitHub at https://github.com/JFrogDev/artifactory-user-

plugins/tree/master/netApp/snapCenter.

5 Conclusion

The integration of Artifactory and SnapCenter 2.0 not only gives developers more control over the data

generated during the CI process but also protects the data by enabling consistent checkpoint snapshots.

These snapshots are taken on the data volumes that provide persistent storage to Artifactory running on

Docker containers, and are instantaneous, thin provisioned, and highly efficient.

When SnapCenter 2.0 creates the thin-provisioned snapshots, developers can clone the data instantly to

test the applications during user acceptance validation and staging. Data can be replicated for disaster

recovery to a remote site or to public hypervisor cloud environments such as Amazon Web Services

(AWS) or Microsoft Azure. During development, developers can use Artifactory as the binary repository

manager on the premises; during deployment, they can use Artifactory in hyperscaler clouds like AWS.

Because SnapCenter and Artifactory integration enables a Data Fabric environment, application and

asset owners can move data to public hyperscaler clouds without vendor lock-in.

https://github.com/JFrogDev/artifactory-user-plugins/tree/master/netApp/snapCenter

https://github.com/JFrogDev/artifactory-user-plugins/tree/master/netApp/snapCenter


References

NetApp SnapCenter datasheet

Appendix

The following shows the content of the Artifactory plug-in, artifactory.json.

JFrog Artifactory installation in a Docker container mounts persistent data volumes by using NetApp

Docker Volume Plugin. The following shows the content of the docker-compose.yml file.

# Version 2 without namespacing. Required env variables :

# ART_LICENSES

# ART_LOGIN (optional, default=admin)

# ART_PASSWORD (optional, default=password)

#

version: '2'

services:

artifactory-node1:

image: jfrog-docker-reg2.bintray.io/jfrog/artifactory-ha-primary:5.1.4

ports:

- 8081:8081

- 10017:10017

volumes:

- ~/data/artifactory/node1/etc:/var/opt/jfrog/artifactory/etc

- artdata:/var/opt/jfrog/artifactory/data

- ~/data/artifactory/node1/logs:/var/opt/jfrog/artifactory/logs

environment:

- DB_TYPE=mysql

- DB_HOST=10.192.39.26

- DB_PORT=3306

- HA_IS_PRIMARY=true

- HA_MEMBERSHIP_PORT=10017

[root@linux-node6 ~]# cat artifactory.json

{

"url": "https://10.192.39.22:8146",

"token": "",

"policy": "ART_Policy",

"resourcegroups": "ART_RG"

}

http://www.netapp.com/us/media/ds-3700.pdf


- DB_USER=artifactory

- DB_PASSWORD=Netapp1!

- ART_LICENSES=" "

restart: always

artifactory-node2:

image: jfrog-docker-reg2.bintray.io/jfrog/artifactory-ha-primary:5.1.4

ports:

- 8082:8081

- 10018:10017

volumes:

- ~/data/artifactory/node2/etc:/var/opt/jfrog/artifactory/etc

- artdata:/var/opt/jfrog/artifactory/data

- ~/data/artifactory/node2/logs:/var/opt/jfrog/artifactory/logs

environment:

- DB_TYPE=mysql

- ART_PRIMARY_BASE_URL=http://artifactory-node1:8081/artifactory

- DB_HOST=10.192.39.26

- DB_PORT=3306

- HA_IS_PRIMARY=false

- HA_MEMBERSHIP_PORT=10017

- DB_USER=artifactory

- DB_PASSWORD=Netapp1!

- ART_LICENSES=" "

depends_on:

- artifactory-node1

restart: always

nginx:

image: jfrog-docker-reg2.bintray.io/jfrog/nginx-art:5.1.4

ports:

- "80:80"

- "443:443"

- "5000-5010:5000-5010"

environment:

- ART_PRIMARY_BASE_URL=http://artifactory-node1:8081/artifactory

volumes:

artdata:

driver: netapp

driver_opts:

snapshotDir: "false"


Refer to the Interoperability Matrix Tool (IMT) on the NetApp Support site to validate that the exact

product and feature versions described in this document are supported for your specific environment. The

NetApp IMT defines the product components and versions that can be used to construct configurations

that are supported by NetApp. Specific results depend on each customer’s installation in accordance with

published specifications.

Copyright Information

Copyright © 2017 NetApp, Inc. All rights reserved. Printed in the U.S. No part of this document covered by copyright may be reproduced in any form or by any means—graphic, electronic, or mechanical, including photocopying, recording, taping, or storage in an electronic retrieval system—without prior written permission of the copyright owner.

Software derived from copyrighted NetApp material is subject to the following license and disclaimer:

THIS SOFTWARE IS PROVIDED BY NETAPP “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL NETAPP BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

NetApp reserves the right to change any products described herein at any time, and without notice. NetApp assumes no responsibility or liability arising from the use of products described herein, except as expressly agreed to in writing by NetApp. The use or purchase of this product does not convey a license under any patent rights, trademark rights, or any other intellectual property rights of NetApp.

The product described in this manual may be protected by one or more U.S. patents, foreign patents, or pending applications.

RESTRICTED RIGHTS LEGEND: Use, duplication, or disclosure by the government is subject to restrictions as set forth in subparagraph (c)(1)(ii) of the Rights in Technical Data and Computer Software clause at DFARS 252.277-7103 (October 1988) and FAR 52-227-19 (June 1987).

Trademark Information

NETAPP, the NETAPP logo, and the marks listed at http://www.netapp.com/TM are trademarks of NetApp, Inc. Other company and product names may be trademarks of their respective owners.

http://mysupport.netapp.com/matrix

http://www.netapp.com/TM

Data Protection for JFrog Artifactory on ONTAP 9 with ... · Data Protection for JFrog Artifactory...

Documents

Transcript of Data Protection for JFrog Artifactory on ONTAP 9 with ... · Data Protection for JFrog Artifactory...