Medical Systems Co. Ltd
LIQUIBASEDBMS Migration Tools for Java
This is a controlled document. Unauthorized access, copying, replication or usage for a purpose other than for which it is intended, are prohibited. All tradthe document have been used for identification purposes only and belong to their respective companies.
Medical Systems Co. Ltd
IQUIBASE 3.5.3DBMS Migration Tools for Java
Version: 0.0.01 (Based on MariaDB 10.1.28)
This is a controlled document. Unauthorized access, copying, replication or usage for a purpose other than for which it is intended, are prohibited. All trademarks that appear in the document have been used for identification purposes only and belong to their
August, 2018
Medical Systems Co. Ltd
3.5.3 DBMS Migration Tools for Java
This is a controlled document. Unauthorized access, copying, replication or usage for a emarks that appear in
the document have been used for identification purposes only and belong to their
5 | P a g e
Table of Contents I. Document Information 7 II. Document Release Note 8 III. Notations Used 9
Overview 10
Database Creation & Import 12
Generate the Change Log 13
1.1 Change Set for Regions Table 15 1.2 Change set to load data of Departments Table (dev, qac) 16 1.3 Change set to load data of Departments Table (uat, pro) 16 1.4 Change set for Primary Key 16 1.5 Change set for Unique Key 17 1.6 Change set for Index 17 1.7 Change set for Foreign Key Constraint 17
Fresher Empty Database Migration 18
Existent Database Fresher Migration 20
Existent Database Update Migration 21
Routine Migration for Development 22
Generate Differential Change Log 23
Exercise & Experiment of Migration 25
Precautions & Good Practice 26
7 | P a g e
Document Information
Issued by: Medical Systems Co. Ltd Head Office: K.S.A - Riyadh FAX: +966-1-414 3211 TEL: +966-1-4141 330 Email: [email protected]
Service-related contact information worldwide
All service-related contact information is available on official
website. Please follow the link: http://medisys.com.my/ihe
8 | P a g e
Document Release Note
Document Details
Details Revision How to Implement Liquibase Migration for Java 0.0.01
Author
Revision Changes Documented Revised Updated 0.0.00 Initial Release Md. MizanurRahman Md Shahed Hossain Aug 16, 2018 0.0.01 Initial Release Md. MizanurRahman Md Shahed Hossain Aug 28, 2018
9 | P a g e
Notations Used
S/N Notation Elaboration 1 DBMS Database Management System 2 RDBMS Relational Database Management System 3 DDL Data Definition Language 4 DML Data Manipulation Language 5 TAB Table 6 NDX Index 7 PKC Primary Key Constraint 8 FKC Foreign Key Constraint 9 UKC Unique Key Constraint
10 SQN Sequence 11 PRO Procedure, Production 12 DEV Development 13 QAC Quality Assurance & Control 14 UAT User Acceptance Testing 15 VEW View 16 INI Initialization 17 TRX Transaction 18 INS Insert 19 UPD Update 20 COL Column
10 | P a g e
Overview Liquibase is an open source community project for refactoring Relational Database Management System. It’s also called VCS (Version Controlling System) for Database. Its platform independent migration tools regardless RDBMS. Only you have to maintain some good practice.
During the migration process running, be aware about contexts those are directly belongs to the deployment environments as following:
1. DEV (Development ) 2. QAC (Quality Assurance & Control) 3. UAT (User Acceptance Test) 4. PRO (Production)
There are some labels; those are usages for filtering the changes logs as following:
1. M00, DDL, TAB (Create/Drop Table) 2. M00, DDL, COL (Add/Drop Column) 3. M00, DDL, VEW (Create/Drop View) 4. M00, DML, INI (Initialization Data) 5. M00, DML, TRX (Transactional Data) 6. M00, DDL, PKC (Primary Key Constraint) 7. M00, DDL, UKC (Unique Key Constraint) 8. M00, DDL, FKC (Foreign Key Constraint) 9. M00, DDL, NDX (Create/Drop Index)
There are some tags those are usages for checkout specific Commit or Rollback. Although Liquibase is a database refactoring and version control system where its performance & success depends on some good practices. Its means order of change log id, dbms, context, labels, logicalFilePath and relativeToChangelogFile. Please find the example project in Github.
# https://github.com/medisysco/medisys-dbms-change
git clone https://github.com/medisysco/medisys-dbms-change.git
cd ./medisys-dbms-change
Snippet: Clone Human Resource Example Project
11 | P a g e
Find the example resources inMySQL/MariaDB Schema files as following
Figure: Liquibase Migration Resources
mkdir -p $HOME/.hmis/etc/dbms/liquibase
cp ./src/test/resources/liquibase/*.properties
cd $HOME/.hmis/etc/dbms/liquibase
explorer .
Snippet: Copy & Edit Properties Files
export MYSQL_HOME=/c/opt/xampp/mysql
export PATH=$PATH:$MYSQL_HOME
Snippet: MySQL Path Settings
inside the project folder. Please find the Liquibaseas following folders:
Resources
/.hmis/etc/dbms/liquibase
test/resources/liquibase/*.properties $HOME/.hmis/etc/dbms/liquibase
/.hmis/etc/dbms/liquibase
Edit Properties Files
/c/opt/xampp/mysql
$MYSQL_HOME/bin
iquibase properties and
/.hmis/etc/dbms/liquibase
12 | P a g e
Database Creation & Import
# execute database creation script for empty password
mysql -uroot < ./src/test/resources/schema/hr_all_ini.sql
# execute database creation script for prompt password in cli
mysql -uroot -p < ./src/test/resources/schema/hr_all_ini.sql
Snippet: Execute Database Creation Dump
# import hr_dev_bas_ini.sql dump into hr_dev from cli
mysql -uroot hr_dev < ./src/test/resources/schema/hr_dev_bas_ini.sql
# import hr_dev_bas_ini.sql dump into hr_dev & prompt password from cli
mysql -uroot -p hr_dev < ./src/test/resources/schema/hr_dev_bas_ini.sql
Snippet: Import Human Resources Development Dump After execution that scripts right now we have Four Databases in MySQL/MariaDB under Root Schema. Those are as following:
1. HR_DEV (Human Resources Development) 2. HR_QAC (Human Resources Quality Assurance & Control) 3. HR_UAT (Human Resources User Acceptance Test) 4. HR_PRO (Human Resources Production)
Where HR_DEV contains some Database Objects and Data, those are used to realizing different environment and context. Please find the Human Resources Example Database structure in the next page.
13 | P a g e
Generate the Change Log
Figure: Human Resources Development Database Considering that we have a Human Resources have to generate change logs then organized the change logs according to the good practise and the company policy. During the time ofbut need to add/overriderelativeToChangelogFile and etc. Where need to specify some initialization/setup data required for production perspectives of development Considering such situation we have defined four contextpro. There are some labels too; those were Considering that, we are going to implement dev.properties as following for respective database
vim $HOME/.hmis/etc/dbms/liquibase/
Snippet: Configure dev.properties
Change Log
Human Resources Development Database
uman Resources Database in development. Regarding this scenario we have to generate change logs then organized the change logs according to the good practise and the
During the time of change log generation we can specify the add/override some id, dbms, context, labels, file,
etc.
Where need to specify some DDL and DML for different context/environment.data required for production but no transactional data required. In
and quality assurance we need to add some Considering such situation we have defined four contexts those are as respectively
o; those were mentioned on page-10.
Considering that, we are going to implement Liquibase migration for fist time. Please for respective database.
/.hmis/etc/dbms/liquibase/dev.properties
properties
. Regarding this scenario we have to generate change logs then organized the change logs according to the good practise and the
we can specify the context and author file, logicalFilePath,
environment. For example data required. In the
some dummy data. those are as respectively dev, qac, uat and
migration for fist time. Please edit the
14 | P a g e
mvn clean install –Pdev,log
Snippet: Liquibase Change log Generation Here is the Change Log generated with exported data. Data may require for Development (dev), Quality Assurance & Control (qac), User Acceptance Test (uat) and Production (pro). Please find the change log in the following location including CSV files those contains data.
# target/db.changelog-V${version.medisys.version}_${change.on}.xml
vim target/db.changelog-V0.0.00_20181010_101010.xml
ls -la target/20181010_101010
Snippet: Change log with data This is the very raw change log those are not production ready. Need to split and amend it according to our organization business. According to the development to production we have four contexts and lots of labels. This is the right time to organize the change log and the data. Don’t mixing the change set and data with other context. Be aware about the good practice during the organization or reorder the change set and the data. Right now change log are ready for development and deployment environment. If you are aware most may have doubt why change log ready for development. No doubt on deployment. Those who have doubt, we would like to appreciate you for raising this doubt. Those who have not any doubt please concentrates the next scenario. Each time of Liquibase update or migration (deprecated) the Liquibase engine compare the change set with databasechangelog table. No such table found in your database, nothing to be wonder and it the reality. If databasechangelog & databasechangeloglock tables are not found, it create itself to perform update where its might be causes error. As Liquibase treated no update performed before. To overcome such type of situation we have to synchronize the change log change set with development and deployment environment for base version only. Since the next version development will be synchronize only and deployment will be updated.
15 | P a g e
Figure: Managed and Ordered Change log
<changeSet author="medisys"
id="1533121211000-7" logicalFilePath=
<createTable tableName=
<column name="region_id"
<constraints nullable=
</column>
<column name="region_name"
</createTable>
</changeSet>
Snippet: Change Set for Regions Table
Managed and Ordered Change log
" labels="m00,ddl,tab" context="dev,qac,uat,pro
logicalFilePath="db.changelog-V0.0.00_20180801_110011.xml
tableName="regions">
"region_id" type="DECIMAL(22)">
nullable="false"/>
"region_name" type="VARCHAR(25)"/>
Regions Table
dev,qac,uat,pro"
V0.0.00_20180801_110011.xml">
16 | P a g e
<changeSet author="medisys" labels="m00,dml,trx" context="dev,qac"
id="1533207611000-2" logicalFilePath="db.changelog-V0.0.00_20180802_110011.xml">
<loadData commentLineStartsWith="#" encoding="UTF-8"
file="20180802_110011/departments__dev_qac.csv" relativeToChangelogFile="true"
quotchar=""" separator="," tableName="departments">
<column header="department_id" name="department_id" type="NUMERIC"/>
<column header="department_name" name="department_name" type="STRING"/>
<column header="manager_id" name="manager_id" type="NUMERIC"/>
<column header="location_id" name="location_id" type="NUMERIC"/>
</loadData>
</changeSet>
Snippet: Change set to load data of Departments Table (dev, qac)
<changeSet author="medisys" labels="m00,dml,ini" context="uat,pro"
id="1533207611000-3" logicalFilePath="db.changelog-V0.0.00_20180802_110011.xml">
<loadData commentLineStartsWith="#" encoding="UTF-8"
file="20180802_110011/departments__uat_pro.csv" relativeToChangelogFile="true"
quotchar=""" separator="," tableName="departments">
<column header="department_id" name="department_id" type="NUMERIC"/>
<column header="department_name" name="department_name" type="STRING"/>
<column header="manager_id" name="manager_id" type="NUMERIC"/>
<column header="location_id" name="location_id" type="NUMERIC"/>
</loadData>
</changeSet>
Snippet: Change set to load data of Departments Table (uat, pro)
<changeSet author="medisys" labels="m00,ddl,pkc" context="dev,qac,uat,pro"
id="1533294011000-2" logicalFilePath="db.changelog-V0.0.00_20180803_110011.xml">
<addPrimaryKey columnNames="department_id" constraintName="PRIMARY"
tableName="departments"/>
</changeSet>
Snippet: Change set for Primary Key
17 | P a g e
<changeSet author="medisys" labels="m00,ddl,ukc" context="dev,qac,uat,pro"
id="1533380411000-1" logicalFilePath="db.changelog-V0.0.00_20180804_110011.xml">
<addUniqueConstraint columnNames="email" constraintName="emp_email_uk"
tableName="employees"/>
</changeSet>
Snippet: Change set for Unique Key
<changeSet author="medisys" labels="m00,ddl,ndx" context="dev,qac,uat,pro"
id="1533466811000-2" logicalFilePath="db.changelog-V0.0.00_20180805_110011.xml">
<createIndex indexName="dept_location_ix" tableName="departments">
<column name="location_id"/>
</createIndex>
</changeSet>
Snippet: Change set for Index
<changeSet author="medisys" labels="m00,ddl,fkc" context="dev,qac,uat,pro"
id="1533553211000-2" logicalFilePath="db.changelog-V0.0.00_20180806_110011.xml">
<addForeignKeyConstraint baseColumnNames="location_id"
baseTableName="departments" constraintName="dept_loc_fk" deferrable="false"
initiallyDeferred="false" onDelete="NO ACTION" onUpdate="NO ACTION"
referencedColumnNames="location_id" referencedTableName="locations"/>
</changeSet>
Snippet: Change set for Foreign Key Constraint In this perspective we have some change logs; those are organized in specific order. Each change set we have already amended some attribute like id, dbms, context, labels, logicalFilePath, relativeToChangelogFile and etc. Right now we need to synchronize the change logs with development Database.
mvn clean install –Pdev,syn
Snippet: Liquibase Change logs Synchronization After having synchronization we are going to update the change logs. If there is an error occurred it’s suppose to that your change logs not ready for deployment else it’s ready.
18 | P a g e
mvn clean install –Pdev,upd
Snippet: Liquibase Change logs Update There are some thumb rules for deployment change logs into Database. Need to be aware and good practise before deployment. We have to consider the following phase in perspective of our database. Those are as following:
1. Fresher Empty Database Migration 2. Existent Database Fresher Migration 3. Existent Database Update Migration
Before migration/update please detect your database phase. Then follow the procedure according to the detection of phase and context. We are going to describe each phase for respective context (Mentioned in Page-10).
Fresher Empty Database Migration
Development:
vim $HOME/.hmis/etc/dbms/liquibase/dev.properties
Snippet: Configure dev.properties
git pull origin master
mvn clean install –Pdev,syn
mvn clean install –Pdev,upd
Snippet: Migration to the Development
19 | P a g e
Quality Assurance & Control:
vim $HOME/.hmis/etc/dbms/liquibase/qac.properties
Snippet: Configure qac.properties
git pull origin master
mvn clean install –Pqac,syn
mvn clean install –Pqac,upd
Snippet: Migration to the QAC User Acceptance Test:
vim $HOME/.hmis/etc/dbms/liquibase/uat.properties
Snippet: Configure uat.properties
git pull origin master
mvn clean install –Puat,syn
mvn clean install –Puat,upd
Snippet: Migration to the UAT Production:
vim $HOME/.hmis/etc/dbms/liquibase/pro.properties
Snippet: Configure pro.properties
git pull origin master
mvn clean install –Ppro,syn
mvn clean install –Ppro,upd
Snippet: Migration to the Production
20 | P a g e
Existent Database Fresher Migration
Development:
git pull origin master
mvn clean install –Pdev,syn
mvn clean install –Pdev,upd
Snippet: Migration to the Development Quality Assurance & Control:
git pull origin master
mvn clean install –Pqac,syn
mvn clean install –Pqac,upd
Snippet: Migration to the QAC User Acceptance Test:
git pull origin master
mvn clean install –Puat,syn
mvn clean install –Puat,upd
Snippet: Migration to the UAT Production:
git pull origin master
mvn clean install –Ppro,syn
mvn clean install –Ppro,upd
Snippet: Migration to the Production
21 | P a g e
Existent Database Update Migration Development:
git pull origin master
mvn clean install –Pdev,syn
mvn clean install –Pdev,upd
Snippet: Migration to the Development Quality Assurance & Control:
git pull origin master
mvn clean install –Pqac,syn
mvn clean install –Pqac,upd
Snippet: Migration to the Quality Assurance & Control User Acceptance Test:
git pull origin master
mvn clean install –Puat,syn
mvn clean install –Puat,upd
Snippet: Migration to the User Acceptance Test Production:
git pull origin master
mvn clean install –Ppro,syn
mvn clean install –Ppro,upd
Snippet: Migration to the Production
22 | P a g e
Routine Migration of Development
In this point we will focus only Development context/environment of respective developer in his own workstation as because here is the matter of aware, need to be concern and have some good practise. Else it might be wastage of time and money. Might be causes the reinvention of this circle what’s already practising by us? Rest of the contexts/environments will be following the thumb rules according to the detection of its phase as mentioned in Pages-18, 19, 20, 21. Detect your migration phase and context then execute the respective command. Let’s start our Daily/Routine Migration for Development in respect of our personal official workstation. Please follow the following step before adding any database object, generating, manipulating, commit and push change logs:
1. Communicate with your team if anybody has change logs to push 2. Before push and pull negotiate with team to overcome possible conflict 3. Always pull first before push, to avoid conflict in change log. 4. After pull resolve the possible conflict with change log.
a. In case of conflict backup, drop the conflicted objects & logs b. Then update your database with newly pulled change logs
mvn clean install -Pdev,upd c. Recreate your database object from backup
5. For conflict free environment please follow the instruction
a. Just backup and drop the generated & manipulated logs b. Update your database with newly pulled change logs
mvn clean install -Pdev,upd
6. Generate and manipulate the change logs using
a. mvn clean install –Pdif b. mvn clean install -Pdev,log
7. Synchronize the newly manipulated change logs
a. mvn clean install –Pdev,syn
8. After Synchronize please Update to cross check a. mvn clean install -Pdev,upd
9. Based on the permission of your team push to git
a. git push –u origin master
23 | P a g e
Generate
Figure: Differences between Development and Production
export MYSQL_HOME=/c/opt/xampp/mysql
export PATH=$PATH:$MYSQL_HOME
Snippet: MySQL Path Settings
# import hr_dev_dif_add.sql
mysql -uroot hr_dev < ./src/test/resources/schema/
# import hr_dev_dif_add.sql
mysql -uroot -p hr_dev <
Snippet: Simulate the Difference
Development:
vim $HOME/.hmis/etc/dbms/liquibase/
Snippet: Configure dif.properties
Generate Differential Change Log
Differences between Development and Production
/c/opt/xampp/mysql
$MYSQL_HOME/bin
.sql dump into hr_dev from cli
./src/test/resources/schema/hr_dev_dif_add
.sql dump into hr_dev & prompt password
< ./src/test/resources/schema/hr_dev_dif_add
ce of Human Resources Development Database
/.hmis/etc/dbms/liquibase/dif.properties
properties
Change Log
hr_dev_dif_add.sql
password from cli
hr_dev_dif_add.sql
24 | P a g e
mvn clean install –Pdif
Snippet: Generate the Differ Change log
# target/db.changelog-V${version.medisys.version}_${change.on}.xml
vim target/db.changelog-V0.0.00_20181010_101010.xml
Snippet: Generated differ change log
Please find the differential change set inside the change log. This is the very raw change log those are not production ready. Need to split and amend it according to our organization business. According to the development to production we have four contexts and lots of labels. This is the right time to organize the change log and the data. Don’t mixing the change set and data with other context. Be aware about the good practice during the organization or reorder the change set and the data. After reverting to the Database Objects Changes we can examine the generated change set inside the change log. Do you distinguish the different of change log and change set?
# import hr_dev_dif_rev.sql dump into hr_dev from cli
mysql -uroot hr_dev < ./src/test/resources/schema/hr_dev_dif_rev.sql
# import hr_dev_dif_rev.sql dump into hr_dev & prompt password from cli
mysql -uroot -p hr_dev < ./src/test/resources/schema/hr_dev_dif_rev.sql
Snippet: Revert the Human Resources Development Database
mvn clean install –Pdif
Snippet: Generate the Differ Change log
# target/db.changelog-V${version.medisys.version}_${change.on}.xml
vim target/db.changelog-V0.0.00_20181010_101010.xml
Snippet: Generated differ change log
25 | P a g e
Exercise & Experiment of Migration Base on the previous exercise right now you are suppose to be experience in Liquibased migration. Most of us naturally talent enough to predict the scenario what would be happened during the migration process mistaken.
1. During the Base Version Change logs Generation (DEV): Considering that there is no databasechangelog & databasechangeloglock tables exist in your Development Database. Change logs generated and manipulated as the organizational perspective for Base Version. Let Update Development Database with the change logs. Examine the update; find out the fault and solution? mvn clean install -Pdev,upd
2. Fresher Empty Database Migration (QAC): Considering this phase let execute the following command and examine what happened? Find out the fault and solution? mvn clean install -Pqac,syn mvn clean install -Pqac,upd
3. Existent Database Fresher Migration (UAT): Considering this phase let execute the following command and examine what happened? Find out the fault and solution? mvn clean install -Puat,upd
4. Existent Database Update Migration (PRO): Considering this phase let execute the following command and examine what happened? Find out the fault and solution? mvn clean install -Ppro,syn mvn clean install -Ppro,upd
26 | P a g e
Precautions & Good Practice
1. Do not add generated change log or change set directly in the change logs 2. Amend those attribute for deployment. Ex: id, dbms, context, labels, logicalFilePath,
relativeToChangelogFile and etc 3. Maintain the order of change log and change set. 4. Separate the context specific change set with Data 5. Be aware about context; do not mix the change log with others 6. Add change set for tag at end of each change log 7. Developer should aware about pull, update, differ and synchronization 8. Before aware to add change set into change log and its effect 9. Don’t use loadUpdateData on before Primary Key creation or implementation. Might raise
data duplication error! In such case use loadData only. Where it should be distinguish by respective context.
10. You should use one of four contexts during update; 11. Empty context Update might cause data duplication and violation of integrity. 12. Example Project: https://github.com/medisysco/medisys-dbms-change
Top Related