Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer....
Transcript of Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer....
![Page 1: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/1.jpg)
Jaywalking in TrafficSafe Migrations at Scale
Brad UraniStaff Engineer
![Page 2: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/2.jpg)
What is Scale?20,000,000 rows fetched / sec
30,000 transactions / sec6 TB
![Page 3: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/3.jpg)
![Page 4: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/4.jpg)
+
+
People
![Page 5: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/5.jpg)
14 Squads working on one of the biggest Rails apps in the world
![Page 6: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/6.jpg)
Goals● 5-10 Deploys per day● Web devs are responsible for their own schema
changes● Minimal involvement by the Site Reliability squad
![Page 7: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/7.jpg)
● Add / Remove Tables● Add / Remove Columns● Adding Constraints (unique, not null, FK)● Add Indexes● Adding Defaults● Moving Data
@BradUrani
![Page 8: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/8.jpg)
> rails generate migration create_products
@BradUrani
![Page 9: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/9.jpg)
class CreateProducts < ActiveRecord::Migration
def change
create_table :products do |t|
t.string :name
t.text :description
t.timestamps
end
end
end
@BradUrani
![Page 10: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/10.jpg)
![Page 11: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/11.jpg)
> rake db:migrate
@BradUrani
![Page 12: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/12.jpg)
![Page 13: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/13.jpg)
class AddTableUsers < ActiveRecord::Migration
def up
add_table :users do |t|
t.string :name, null: false
t.string :email, null: false
end
add_index :users, :email, unique: true
end
def down
remove_table :users
end
end
@BradUrani
![Page 14: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/14.jpg)
class AddColumnIsAdmin < ActiveRecord::Migration
def up
add_column :users, :is_admin, :boolean, null: false, default: false
end
def down
remove_column :users, :is_admin
end
end
@BradUrani
![Page 15: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/15.jpg)
Is it safe?
![Page 16: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/16.jpg)
![Page 17: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/17.jpg)
![Page 18: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/18.jpg)
![Page 19: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/19.jpg)
☎
@BradUrani
![Page 20: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/20.jpg)
https://www.postgresql.org/docs/9.1/static/explicit-locking.html
![Page 21: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/21.jpg)
2 Ways to fail
● Waiting for lock● During Query
@BradUrani
![Page 22: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/22.jpg)
class AddColumnIsAdmin < ActiveRecord::Migration
def up
execute "SET LOCAL lock_timeout = '5s'"
add_column :users, :is_admin, :boolean, null: false, default: false
end
def down
remove_column :users, :is_admin
end
end
@BradUrani
![Page 23: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/23.jpg)
TIMEOUT!!
![Page 24: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/24.jpg)
https://github.com/procore/handcuffs
![Page 25: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/25.jpg)
# config/initializers/handcuffs
Handcuffs.configure do |config|
config.phases = [:pre_restart, :downtime, :post_restart]
config.default_phase = :pre_restart
end
@BradUrani
![Page 26: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/26.jpg)
rake handcuffs:migrate[pre_restart]
rake handcuffs:migrate[downtime]
rake handcuffs:migrate[post_restart]
rake handcuffs:migrate[all]
@BradUrani
![Page 27: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/27.jpg)
![Page 28: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/28.jpg)
A Multi-Part Migration Strategy
@BradUrani
![Page 29: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/29.jpg)
class AddColumnIsAdmin < ActiveRecord::Migration
def up
add_column :users, :is_admin, :boolean
end
end
![Page 30: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/30.jpg)
class BackfillUserIsAdmin < ActiveRecord::Migration
phase :post_restart
def up
User.update_all(is_admin: false)
end
end
![Page 31: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/31.jpg)
![Page 32: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/32.jpg)
class BackfillUserIsAdmin < ActiveRecord::Migration
phase :post_restart
disable_ddl_transaction!
def up
User.find_in_batches(batch_size: 1000) do |users|
User.where(id: users.map(&:id)).update_all(is_admin: false)
end
end
end
![Page 33: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/33.jpg)
● Still nullable● Still no default value
Potential Problems
@BradUrani
![Page 34: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/34.jpg)
Race conditions!!!!
![Page 35: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/35.jpg)
class BackfillWithLockAddConstraints < ActiveRecord::Migration
phase :post_restart
def up
execute <<-SQL
LOCK TABLE users IN ACCESS EXCLUSIVE MODE;
SQL
User.where(is_admin: nil).update_all(is_admin: false)
change_column_default :users, :is_admin, false
change_column_null :users, :is_admin, false
end
end
![Page 36: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/36.jpg)
![Page 37: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/37.jpg)
● ACCESS EXCLUSIVE lock prevents race conditions● Changing to non-null with default doesn't change that much● post_restart minimizes time when code / db differ
@BradUrani
![Page 38: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/38.jpg)
5 Part Deploy
![Page 39: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/39.jpg)
class AddColumnIsAdmin < ActiveRecord::Migration
def up
add_column :users, :is_admin, :boolean
end
end
![Page 40: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/40.jpg)
class BackfillUserIsAdmin < ActiveRecord::Migration
phase :post_restart
disable_ddl_transaction!
def up
User.find_in_batches(batch_size: 1000) do |foos|
User.where(id: user.map(&:id)).update_all(is_admin: false)
end
end
end
![Page 41: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/41.jpg)
class CreateOptionalIndex < ActiveRecord::Migration
phase :post_restart
disable_ddl_transaction!
def change
add_index :users,
:is_admin,
where: "is_admin IS NULL",
algorithm: :concurrently
end
end
![Page 42: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/42.jpg)
class BackfillWithLockAddConstraints < ActiveRecord::Migration
phase :post_restart
def up
execute <<-SQL
LOCK TABLE users IN ACCESS EXCLUSIVE MODE;
SQL
User.where(is_admin: nil).update_all(is_admin: false)
change_column_default :users, :is_admin, false
change_column_null :users, :is_admin, false
end
def down
change_column_null :users, :is_admin, true
change_column_default :users, :is_admin, nil
end
end
![Page 43: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/43.jpg)
class DropOptionalIndex < ActiveRecord::Migration
phase :post_restart
disable_ddl_transaction!
def change
remove_index :users,
:is_admin,
where: "is_admin IS NULL",
algorithm: :concurrently
end
end
![Page 44: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/44.jpg)
Multiple Deploys
@BradUrani
![Page 45: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/45.jpg)
Rename:
1. Create a new column
2. Create code that writes to both columns
3. Write to both columns
4. Backfill data from the old column to the new column
5. Deploy code that only writes to new column
6. Drop the old column
![Page 46: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/46.jpg)
Operations
@BradUrani
![Page 47: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/47.jpg)
Does it remove something used in old code?
● Can't be pre_restart
Does it add something used by new code
● Can't be post_restart
@BradUrani
![Page 48: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/48.jpg)
Does it lock?
● More than 5 seconds?○ Can it be broken up?
■ Yes● Multi-part pre- and post-restart
■ No● Downtime
● Less than 5 seconds○ Are you sure?
■ Lock timeout
@BradUrani
![Page 49: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/49.jpg)
create_table
@BradUrani
![Page 50: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/50.jpg)
create_table
pre_restart
● Not referenced by old code
@BradUrani
![Page 51: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/51.jpg)
drop_table
@BradUrani
![Page 52: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/52.jpg)
drop_table
post_restart
● Referenced by old code
Or downtime to be safe. Rolling restarts can cause errors
@BradUrani
![Page 53: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/53.jpg)
add_column (nullable, no default)
@BradUrani
![Page 54: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/54.jpg)
add_column (nullable, no default)
pre_restart with timeout
● Not referenced by old code
@BradUrani
![Page 55: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/55.jpg)
add_column (not nullable, with default)
@BradUrani
![Page 56: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/56.jpg)
add_column (not nullable, with default)
downtime
● Or break it up
(unless it's empty)
@BradUrani
![Page 57: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/57.jpg)
remove_column
@BradUrani
![Page 58: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/58.jpg)
remove_column post_restart (as of rails 4)
● Can't run pre_restart because code references it● Can run post_restart because INSERTS don't include all columns
@BradUrani
![Page 59: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/59.jpg)
change_column_null
@BradUrani
![Page 60: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/60.jpg)
change_column_null Non-Nullable ➡� Nullable pre_restart
Nullable ➡� Non-Nullable downtime or multi-part● Must be backfilled or empty● Locks
@BradUrani
![Page 61: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/61.jpg)
change_column_default
@BradUrani
![Page 62: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/62.jpg)
change_column_default Default ➡� No Default post_restart (either really)
No Default ➡� Default pre_deploy● But mostly this is done while making it non-nullable
@BradUrani
![Page 63: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/63.jpg)
change_column
@BradUrani
![Page 64: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/64.jpg)
change_column● Data types?● Validations?
Depends
Err on the side on downtime - every case different
@BradUrani
![Page 65: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/65.jpg)
add_foreign_key
@BradUrani
![Page 66: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/66.jpg)
add_foreign_key downtime
● ACCESS EXCLUSIVE on BOTH sides
Exceptions
● During create_table: pre_restart● During create_column, nullable FK: pre_restart● Be sure there are no violations!!!!
● With no index it will require scan, so you probably want an
index in PK side
@BradUrani
![Page 67: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/67.jpg)
remove_foreign_key
@BradUrani
![Page 68: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/68.jpg)
remove_foreign_key pre_restart
@BradUrani
![Page 69: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/69.jpg)
add_index(not unique)
@BradUrani
![Page 70: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/70.jpg)
add_index(not unique)
post_restart
● Use with disable_ddl_transaction! and algorithm: concurrently
@BradUrani
![Page 71: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/71.jpg)
add_index(unique)
@BradUrani
![Page 72: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/72.jpg)
add_index(unique)
post_restart
● Use with disable_ddl_transaction! and algorithm: concurrently
@BradUrani
![Page 73: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/73.jpg)
Order201606129876_add_table_foo pre_restart
201606234567_add_index_bar post_restart
201606346543_add_move_data post_restart
201606487654_add_table_baz pre_restart
201606587654_add_default post_restart
@BradUrani
![Page 74: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/74.jpg)
Rollback?
@BradUrani
![Page 75: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/75.jpg)
Preserve Autonomy?
● Focus on Training● Focus on Tooling
Tradeoff?
![Page 76: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/76.jpg)
1 Feature, 1 Deploy
● Can't Eliminate all Lock Contention
Tradeoff
![Page 77: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/77.jpg)
Speed
● Requires some risk
Tradeoff
![Page 78: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/78.jpg)
Not one size fits all
![Page 79: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/79.jpg)
Resources● https://blog.codeship.com/rails-migrations-zero-downtime/● https://github.com/procore/handcuffs● https://github.com/ankane/strong_migrations
@BradUrani
![Page 80: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/80.jpg)
Who am I?
linkedin.com/in/bradurani@BradUraniI tweet at Connect with me
My seldomly updated blog
fractalbanana.com
I work in Santa Barbara at
![Page 81: Jaywalking in Traffic - PGCon · 2020-01-04 · Safe Migrations at Scale Brad Urani Staff Engineer. What is Scale? 20,000,000 rows fetched / sec 30,000 transactions / sec 6 TB + +](https://reader034.fdocuments.us/reader034/viewer/2022042417/5f33036743847237452b0edc/html5/thumbnails/81.jpg)