SubversionGuide.docx

SVN & Visual Studio Basics, Naming and Use Standards

Developer’s Guideline

1/6/2016 4:11 PM Subversion Guide 2016 Page: 1

Topic Page

SVN & Visual Studio Basics

Installation and Access

Local Machine Setup

Working a JIRA Ticket

Local Machine Has Current Code

Developer is Ready to Make Changes

Peer Review and Tagging in SVN to Prepare for Promotion

Development Work Completed

If Adding or Modifying SSIS Packages

If Adding or Modifying a SQL Agent job

Example of link or script to do the If Adding or Modifying a SQL Agent job

Peer Review

Tag the Ticket into Code Review Approved

Name and Use Standard

RELEASE_NOTES

SCRIPTS

SQL_AGENT_JOBS

SSIS_PROJECT

Promotion Process – Code Review


Providing documentation to assist developers using Subversion is an extremely important focus for us all. As part of this focus, we are developing this document, the Subversion Guide. The Subversion Guide includes the basic information needed to use the tool as well as naming and use standards that should be used during the process. This is a living document and will change as we learn more about our process and challenges.

Installation and Access Before you can use Subversion you must install the software on your local machine and gain access as a user of the software: 1. Install of software

a. You can find the install file here

a. \\corp.twcable.com\west\_Legacy\LOSORANFILE01\GG_Share\TechOps\Applications\

b. The file name is TortoiseSVN1.8.8.25755x64svn1.8.10.msi

2. Access to use (if you are new member to BI Tech OPS group) a. Contact Daniel Bowlin or Shashi Kadambi to assist.

Local Machine Setup

1. Start out by creating a folder named SVN in the root of your C drive or somewhere else you feel is appropriate.

2. Inside that SVN folder create a folder for each TechOps project

a. BI_TOPS_DIRECT_OUTPUT

b. BI_TOPS_LOADS

c. BI_TOPS_MARKETING

d. BI_TOPS_METRICS


e. BI_TOPS_PMT

f. BI_TOPS_REALTIME

g. BI_TOPS_SBDB

h. BI_TOPS_SELF_BILLING

i. BI_TOPS_UTILITIES

j. BI_TOPS_WFDB

k. BI_TOPS_WFTXDB

l. BI_TOPS_WOA

m. BI_TOPS_WOBCDB

n. BI_TOPS_WODB

o. BI_TOPS_WORK


Working a JIRA Ticket

3. When you have a new ticket for work in one of these projects go inside the project folder and create a folder with the ticket name (i.e. BI_TOPS_UTILITIES/IBTO1144)

Important note: The majority of you would work in the BI_TOPS_LOADS or

BI_TOPS_WODB projects.

4. Next right click the folder and choose TortoiseSVN > RepoBrowser


a. Put in the path to the project you are working on, to the branches folder.

Important note: SVN is case sensitive; please make sure all naming is done in the proper

CASE. Be aware that the project can be different every time.

b. Click OK.

5. In the Repobrowser, you need to create a branch for your ticket.

a. Right click the trunk folder and choose Copy To


b. Enter the path to the branch and the ticket number you are creating.

Important note: The “branches” should be lower case, and the ticket number should be upper case and include the dash.

c. Click OK and put in a message that includes the ticket number and anything else you want.


d. Click OK and the new branch for your ticket will be created. You may need to refresh the

branches a few times to see it, or close and reopen the Repobrowser.

A suggested way to refresh: Collapse branches, right click, then refresh, then expand

branch to see new folder

e. Once you can see the new branch right click on it and choose Checkout to create a copy

on your computer.


Important note: On the checkout dialog, make sure the Checkout directory is the one you

created on your computer.


f. Click OK and after the checkout is finished, click OK again.

g. Click Ok.

Local Machine Has Current Code

Now your local folder has a copy of the current code.

If you are modifying SSIS packages make sure you open the project from your

SVN/SSIS_Project folder/Ticket_Name.

Each ticket should have its own code repository on both the server and your local machine.

It is important to work each ticket code separately when you open it in SSDT.


Developer is Ready to Make Changes

Developer now opens Visual Studio to make changes.

Open the project:

Browse to the SVN/SSIS_Project/Ticket_Name/SSIS_PROJECT folder (i.e.

SVN/BI_TOPS_loads/ibto1212/SSIS_PROJECT/TOPSLoads.sln)


6. Once you have finished your work for the day, or when the entire change is complete, save and

build the project as you normally would.

7. Then go back to the file system and you will notice that the folder has a red exclamation indicating

that something has changed. Right click on the folder with the ticket number and choose SVN

Commit… to push the changes back up to the branch in SVN.


It’s a good policy to update SVN at least every day you make changes. This way if you are out

for an extended period, someone else can pick up the most current version of your changes

and begin work from there.

a. Put in a message.

Hint, click Recent Messages and use the same base message every time.


b. Click the ALL link (All is the default) and then click OK and your code changes should be

pushed back up to SVN. Click OK to close the Commit dialog.

Peer Review and Tagging in SVN to Prepare for Promotion

Development Work Completed

When all development work is completed, and you believe you are ready for peer review, make sure

you COMMIT your work to SVN.

If Adding or Modifying SSIS Packages If you are adding or modifying SSIS packages, please make sure to note in the Jira ticket or in the

release notes folder in SVN, each package that has been added or modified.

It is suggested: Developers put release or change notes right in SVN, then there is the history of

changes right in the source control. This will be important during the promotion process.


If Adding or Modifying a SQL Agent job If you going to add or modify a SQL Agent job the best approach is to supply a script for the entire

new version of the job with promotion instructions to rename or delete the old job and then execute

the script to create the new version of the job. However, it is not always practical to provide a full

script because of parallel changes to the SQL Agent job. If there is a dependency; setup a

dependency check within the job. If you do not provide a script, you will be expected to provide

explicit instructions including:

1. The name of the new step

2. The type of step it is (SQL, SSIS, etc.)

a. The connection and path to the package if it is SSIS. Making sure to use the

production

servers.

b. The exact script if it is SQL.

3. The environment or database connection to be used, making sure to use the appropriate

production environment because the environments and environment ID numbers are not

the

same between DEV and PROD.

4. Any environment overrides, or package specific parameters or connections.

5. Where exactly in the job where the step should be placed

a. For example: This step should be between the step labeled “start” and the step

labeled “Check HouseDim Load”

Then, using the repobrowser open the SVN branch that contains your work and tag it to

development.

Example of link or script to do the If Adding or Modifying a SQL Agent job


In the address window that opens change “branches” to tags/development and then add a suffix to

the ticket number of “_DEV_COMPLETED_01”.

Note: If you have tagged this work to development before and had to go back and do more work the

last 2 digits of the suffix will change appropriately to reflect the version i.e. 02, 03, 04, etc.


REMINDER: SVN is case sensitive, please observe the case used below in all instances.

Peer Review At this point you are ready for peer review. Schedule that to be done.

Tag the Ticket into Code Review Approved When peer review is complete send a note to the promotions team (currently Dan for BC & Shashi for

Resi) and ask them to tag the ticket into code review approved. We will then tag the ticket something

like this:

After the work is tagged to code review, you can fill out your promotion request document with

hyperlinks to the appropriate SVN objects and attach it to the ticket.

You can then refer to Section II. Ticket Information, Section III. Pending Production Promotions and Section V. Links of the Promotion Process document to start and complete the promotion process per your role on the project.


There are 2 types of SVN projects. An SSIS project, and a database project. An SSIS project in SVN will

have 4 subfolders RELEASE_NOTES, SCRIPTS, SQL_AGENT JOBS, and SSIS_PROJECT. A database

project in SVN will have 2 subfolders RELEASE_NOTES, SCRIPTS. The type of objects inside each

folder and the naming should following these conventions.

RELEASE_NOTES

There can be 2 documents in here. Both are optional, but when used should conform to the following

naming:

1. Development notes document. This may be used this to track the all the changes made to a

project and scripts. This is a simple checklist you can use to remember all the objects you

created or altered during the development process for the ticket.

Operations support opens and closes several projects every day. This is a great way to keep

track of things performed over time along with the notes you may put in the ticket itself.

i. Naming convention TicketName_ReleaseNotes

ii. Example IBTO604_ReleaseNotes

iii. Sample contents of file

1. –added new project parameter parm_DaysAgo

2. –changed package ImportEmplData.dtsx

3. –created alter script stagingHouseDim table

2. Deployment, or promote to production request. This is basically a copy of the promotion

request put in the ticket. Please follow this request for both the document you put in the

ticket itself, and this folder in SVN.

SCRIPTS


All scripts should be kept in database projects in SVN. Please do not use the scripts folder in the SSIS

SVN projects.Then you only have to worry about the new scripts you create.

1. Script naming conventions: ScriptAction_Database_objectname

a. There are a number of ScriptActions you can use

i. CreateSp (create stored procedure)

ii. AlterSp (Alter stored procedure)

iii. DropSp (Drop stored procedure)

iv. CreateVw (create view)

v. AlterVw (Alter view)

vi. DropVw (Drop view)

vii. CreateTbl (create table)

viii. AlterTbl (Alter table)

ix. DropTbl (Drop table)

x. CreateFn (create function)

xi. AlterFn (Alter function)

xii. DropFn (Drop function)

b. Database is an abbreviation

i. SB (dwh_trgt_SelfBilling)

ii. WF (dwh_trgt_workforce)

iii. WFTX (dwh_trgt_workforce_TX)

iv. WO (dwh_trgt_workorders)

v. WOBC (dwh_trgt_workorders_BC)

vi. ICOM (ICOMS_STG)

c. Object name is the actual object name without the schema prefix

d. To alter a stored procedure in the dwh_trgt_workorders database named

spUpdateHouseDim. The script name saved in this folder in SVN would be

AlterSp_WO_ spUpdateHouseDim

SQL_AGENT_JOBS

This is where scripts to create new SQL Agent jobs, or replace existing SQL Agent jobs should be

stored. It is not usually practical to alter a SQL Agent job so I typically drop the old one and create a

new one. So, I would recommend that all scripts here be drop & create scripts and the naming

convention simply be the name of the SQL Agent job following the standards for job naming that we

already use for TOPS jobs.


One additional note here is that new SQL agent jobs should all have a start logging step, and a finish

logging step. This is a process in itself, but if you look at existing jobs, should be fairly

selfexplanatory. Sometimes there are 2 sets of these logging steps in a single job. One set to mark

the start and end of the job itself, and the other to mark the start and end of a specific part of the job

upon which other jobs are dependent. You can ask Shashi or me for help if you need.

Also make sure that any jobs you create have the proper dependency checks set up, and that any

other jobs that will depend on this new job have the proper dependencies set up as well.

SSIS_PROJECT

SSDT will name all the root level objects (project, solution, and subfolders, etc. so there is no need to

worry about that. Unfortunately the individual package naming in TOPS has been fairly inconsistent,

so at this point it will be very difficult to enforce any standard. SSIS Standards to use moving forward

are as follows:

Solution and Project Conventions

● When creating a new package, change the following values in the advanced properties of the

package

o Generate a new package ID

o Change name to reflect the name you gave the package

o Update the Creator name to yours.

● Projects/solutions should be saved in the appropriate project in SVN

● All Solution, Project and Package naming should be in camel case with no spaces. Task and

Transform names can use spaces. Please make all names meaningful to the task they are

performing.

● Please use the following prefixes for your task and transforms

Task/Transform Prefix For Loop Container FLC


Foreach Loop Container FELC Sequence Container SEQC ActiveX Script AXS Analysis Services Execute DDL ASE Analysis Services Processing ASP Bulk Insert BLK Data Flow DFT Data Mining Query DMQ Execute DTS 2000 Package EDPT Execute Package EPT Execute Process EPR Execute SQL ESQL File System FST FTP FTP Message Queue MSMQ Script SCR Send Mail SMT Transfer Database TDB Transfer Error Messages TEM Transfer Jobs TJT Transfer Logins TLT Transfer Master Stored Procedures TSP Transfer SQL Server Objects TSO Web Service WST WMI Data Reader WMID WMI Event Watcher WMIE XML XML DataReader Source DRSRC

Excel Source EXSRC

Flat File Source FFSRC

OLE DB Source OLESRC

Raw File Source RFSRC


XML Source XMLSRC

Aggregate AGG

Audit AUD

Character Map CHM

Conditional Split CSPL

Copy Column CPYC

Data Conversion DCNV

Data Mining Query DMQ

Derived Column DER

Export Column EXPC

Fuzzy Grouping FZG

Fuzzy Lookup FZL

Import Column IMPC

Lookup LKP

Merge MRG

Merge Join MRGJ

Multicast MLT

OLE DB Command CMD

Percentage Sampling PSMP

Pivot PVT

Row Count CNT

Row Sampling RSMP

Script Component SCR

Slowly Changing Dimension SCD

Sort SRT

Term Extraction TEX

Term Lookup TEL

Union All UNA

Unpivot UPVT

Data Mining Model Training DMMTDST

DataReader Destination DRDST

Dimension Processing DPDST


Excel Destination EXDST

Flat File Destination FFDST

OLE DB Destination OLEDST

Partition Processing PPDST

Raw File Destination RFDST

Recordset Destination RSDST

SQL Server Destination SSDST

SQL Server Mobile Destination SSMDST

Packages should be named according to the following standard

● Package name with no spaces or underscores

Connection conventions

● Connections should be named for the database, flat file, etc. only, no hostname

● All connections should use parameters.

Email task conventions

● All components of email should be variable.

o From, To, subject, message, etc.

● All email variables should use configurations

● The package naming convention for the variable is

o First the solution or project name

o Followed by an underscore and the variable name

Documentation conventions

● Task and Transform naming conventions should use the prefix naming conventions above

● Annotation should be added for every task or group of tasks with a common purpose

Variable conventions

● It is a best practice to use variables for as many of the package settings and values as possible.

● Please use camel case and do not use spaces in variable names


Parameter conventions

● All connection parameters should be prefixed by conn_

o i.e conn_dwh_trgt_workorders

● All non connections parameters should be prefixed by parm_

o i.e. parm_DaysAgo

● Parameter names. There are two primary types of parameters in use. Connection

parameters and runtime value parameters. Each starts with a lower case prefix indicating the

type of parameter and then continues with a camel case name. Since SSIS parameters and

variables are case sensitive sticking to these case rules will be important to make sure we set

up SQL Agent job input parameters properly.

● Connection parameters: Please prefix the name with conn_. For example in the existing

TOPSLOAD project there is a connection parameter named conn_DwhTrgtWorkforce that is

the connection string to the dwh_trgt_workforce db. In this case the underscore (_) is used as

a kind of name space separator. With that in mind the conn is separated from the database

name by an underscore to indicate the change from the parameter type (conn) to the

parameter specific name. The underscores are removed from the db name to stay consistent

with the idea of the name space separator.


Promotion Process – Code Review The promotion team will take your promotion document from the ticket, and any release notes

in the ticket or in SVN and use that to move the objects into production.

1. The trunk code will be compared to the new code to confirm all changes are valid, and

will not overwrite any other changes accidentally.

2. All scripts will be executed on the production server as requested in the promotion

document.

3. All SQL agent jobs will be added or modified as requested in the ticket.

a. Promoter must validate the following after adding or altering a job

● The new job or step is showing no errors

● New steps are set to retry at least 5 times at 5 minute intervals if

appropriate.

● Any new jobs or steps must have valid paths to the PRODUCTION server,

global environments, parameters as needed

4. If there are new SSIS packages the promoter will

● Create a local copy of the code review approved version of the code

● Create a local copy of the trunk

● Open the trunk version of the project in SSDT

● Add the altered packages/objects from the code review approved version of

the code using the release notes in the ticket or SVN to the trunk version of

the code

● Verify all connections and parameters in the new objects are not showing

errors after being added to the trunk

● Commit the trunk back to SVN

5. After all the deploy/execute tasks have been done the promoter will tag the coder

review approved code to production MIG_COMPLETED_ with at date suffix YYYYMMD


6. Once the code has been tagged to production, the promoter will go back to the pending

promotions spreadsheet and mark the promotion complete and add any notes as

necessary.


SubversionGuide.docx

Documents

Transcript of SubversionGuide.docx