Parted ways With Partitioning? Software Engineer, Timescale · Partitioning with PostgreSQL 12...

Parted ways With Partitioning?It’s time to reconsider:How PostgreSQL Revived Partitioning to Improve Database Performance

Feike Steenbergen

Software Engineer, Timescale

[email protected] · github.com/timescale

Partitioning

Partitioning?

• Split up a table into multiple parts

Partitioning?

• Split up a table into multiple parts• Increases manageability of large tables

Partitioning?

• Split up a table into multiple parts• Increases manageability of large tables• Can increase performance

Partitioning?

• Split up a table into multiple parts• Increases manageability of large tables• Can increase performance• In a distributed setting: allows horizontal scaling

Partitioning

Time Based Partitioning

Partitioning

Time Based Partitioning

Partitioning

When: 2019, week 31

Time based and Source based partitioning

2-dimensionalPartitioning

Time based and Source based partitioning

2-dimensionalPartitioning

Source: IndustryWhen: 2019, week 31

Partitioning

You need a problem first

Problems

• Very large tables (> 1 TB)

Problems

• Very large tables (> 1 TB)○ Vacuum takes days (every few days)

Problems

• Very large tables (> 1 TB)○ Vacuum takes days (every few days)○ (re)index takes very long

Problems

• Very large tables (> 1 TB)○ Vacuum takes days (every few days)○ (re)index takes very long○ Any incident with this table will be hairy

Problems

• Data deletion policies

Problems

• Data deletion policies○ DELETE takes a very, very long time

Problems

• Data deletion policies○ DELETE takes a very, very long time○ Generates a lot of WAL

Problems

• Data deletion policies○ DELETE takes a very, very long time○ Generates a lot of WAL○ Data is (no longer) clustered

Problems

• Data deletion policies○ DELETE takes a very, very long time○ Generates a lot of WAL○ Data is (no longer) clustered○ Vacuum takes longer

Partitioning

Recent history

PostgreSQL < 10

Use Inheritance

Increased Query Planning Time

More locks needed

You must create new partitions

Some Foreign Key support

Trigger maintenance

Write manual check constraints

Partitions can overlap

Update does not move rows

No shared (unique) index

Inheritance with PostgreSQL 9.6

Decreased Query execution time

Data retention/tiering is fast

Vacuum/analyze/reindex is faster

Partitions can be on a foreign server

PostgreSQL 10

Declarative PartitioningHuge speedup on inserts

More locks needed

No Foreign Key support

Trigger maintenance

Write manual check constraints

Partitions can overlap

Partitioning with PostgreSQL 10

Declarative Partitioning

PostgreSQL 11

Features, features, features

More locks needed

No Foreign Key support*

Hash Partitioning

Partition Pruning at Init/Execute time

PostgreSQL 12

Features and Performance

More locks needed

No Foreign Key support

Hash Partitioning

Partition Pruning at Init/Execute time

Faster COPY support

Catalog improvements (tablespace, pg_indexes)

Full Foreign Key support

Foreign Keys

And why it matters

order_id: int64line_no: int

order_line

order_id: int64customer_id: int64

customer_id: int64

customer

order_line

customer_id: int64

customer

partition 1

partition 2

partition n

partition 2

partition 1

partition n

partition 1

9.6: 1 FK for every partition

order_line

customer_id: int64

customer

partition 1

partition 2

partition n

partition 2

partition 1

partition n

partition 1

10: No Foreign Keys!

order_line

customer_id: int64

customer

partition 1

partition 2

partition n

partition 2

partition 1

partition n

partition 1

11: Foreign Keys from partitioned table to regular table

partition 3

partition 2

order_line

customer_id: int64

customer

partition 1

partition 2

partition n

partition 2

partition 1

partition n

partition 1

12: Full support for foreign keys

partition 3

partition 2

order_line

customer_id: int64

customer

partition 1

partition 2

partition n

partition 2

partition 1

partition n

partition 1

12: Full support for foreign keysThis is great!

Insert performance

Gone are the triggers

IF ( NEW.id >= 24750000 AND NEW.id < 25000000)THEN INSERT INTO hackernews_p0099 VALUES (NEW.*);

-- 98 more IF THEN statements

ELSIF ( NEW.id >= 0 AND NEW.id < 250000)THEN INSERT INTO hackernews_p0000 VALUES (NEW.*);ELSE RAISE EXCEPTION 'Partition hackernews_p% does not yet exist.', NEW.id/250000;END IF;RETURN NULL;

Example trigger

IF ( NEW.id >= 24750000 AND NEW.id < 25000000)THEN INSERT INTO hackernews_p0099 VALUES (NEW.*);ELSIF ( NEW.id >= 24500000 AND NEW.id < 24750000)THEN INSERT INTO hackernews_p0098 VALUES (NEW.*);ELSIF ( NEW.id >= 24250000 AND NEW.id < 24500000)THEN INSERT INTO hackernews_p0097 VALUES (NEW.*);ELSIF ( NEW.id >= 24000000 AND NEW.id < 24250000)THEN INSERT INTO hackernews_p0096 VALUES (NEW.*);ELSIF ( NEW.id >= 23750000 AND NEW.id < 24000000)THEN INSERT INTO hackernews_p0095 VALUES (NEW.*);ELSIF ( NEW.id >= 23500000 AND NEW.id < 23750000)THEN INSERT INTO hackernews_p0094 VALUES (NEW.*);ELSIF ( NEW.id >= 23250000 AND NEW.id < 23500000)THEN INSERT INTO hackernews_p0093 VALUES (NEW.*);ELSIF ( NEW.id >= 23000000 AND NEW.id < 23250000)THEN INSERT INTO hackernews_p0092 VALUES (NEW.*);ELSIF ( NEW.id >= 22750000 AND NEW.id < 23000000)THEN INSERT INTO hackernews_p0091 VALUES (NEW.*);ELSIF ( NEW.id >= 22500000 AND NEW.id < 22750000)THEN INSERT INTO hackernews_p0090 VALUES (NEW.*);ELSIF ( NEW.id >= 22250000 AND NEW.id < 22500000)THEN INSERT INTO hackernews_p0089 VALUES (NEW.*);ELSIF ( NEW.id >= 22000000 AND NEW.id < 22250000)THEN INSERT INTO hackernews_p0088 VALUES (NEW.*);ELSIF ( NEW.id >= 21750000 AND NEW.id < 22000000)THEN INSERT INTO hackernews_p0087 VALUES (NEW.*);ELSIF ( NEW.id >= 21500000 AND NEW.id < 21750000)THEN INSERT INTO hackernews_p0086 VALUES (NEW.*);ELSIF ( NEW.id >= 21250000 AND NEW.id < 21500000)THEN INSERT INTO hackernews_p0085 VALUES (NEW.*);ELSIF ( NEW.id >= 21000000 AND NEW.id < 21250000)THEN INSERT INTO hackernews_p0084 VALUES (NEW.*);ELSIF ( NEW.id >= 20750000 AND NEW.id < 21000000)THEN INSERT INTO hackernews_p0083 VALUES (NEW.*);ELSIF ( NEW.id >= 20500000 AND NEW.id < 20750000)THEN INSERT INTO hackernews_p0082 VALUES (NEW.*);ELSIF ( NEW.id >= 20250000 AND NEW.id < 20500000)THEN INSERT INTO hackernews_p0081 VALUES (NEW.*);ELSIF ( NEW.id >= 20000000 AND NEW.id < 20250000)THEN INSERT INTO hackernews_p0080 VALUES (NEW.*);ELSIF ( NEW.id >= 19750000 AND NEW.id < 20000000)THEN INSERT INTO hackernews_p0079 VALUES (NEW.*);ELSIF ( NEW.id >= 19500000 AND NEW.id < 19750000)THEN INSERT INTO hackernews_p0078 VALUES (NEW.*);ELSIF ( NEW.id >= 19250000 AND NEW.id < 19500000)THEN INSERT INTO hackernews_p0077 VALUES (NEW.*);ELSIF ( NEW.id >= 19000000 AND NEW.id < 19250000)THEN INSERT INTO hackernews_p0076 VALUES (NEW.*);ELSIF ( NEW.id >= 18750000 AND NEW.id < 19000000)THEN INSERT INTO hackernews_p0075 VALUES (NEW.*);ELSIF ( NEW.id >= 18500000 AND NEW.id < 18750000)THEN INSERT INTO hackernews_p0074 VALUES (NEW.*);ELSIF ( NEW.id >= 18250000 AND NEW.id < 18500000)THEN INSERT INTO hackernews_p0073 VALUES (NEW.*);ELSIF ( NEW.id >= 18000000 AND NEW.id < 18250000)THEN INSERT INTO hackernews_p0072 VALUES (NEW.*);ELSIF ( NEW.id >= 17750000 AND NEW.id < 18000000)THEN INSERT INTO hackernews_p0071 VALUES (NEW.*);ELSIF ( NEW.id >= 17500000 AND NEW.id < 17750000)THEN INSERT INTO hackernews_p0070 VALUES (NEW.*);ELSIF ( NEW.id >= 17250000 AND NEW.id < 17500000)THEN INSERT INTO hackernews_p0069 VALUES (NEW.*);ELSIF ( NEW.id >= 17000000 AND NEW.id < 17250000)THEN INSERT INTO hackernews_p0068 VALUES (NEW.*);ELSIF ( NEW.id >= 16750000 AND NEW.id < 17000000)THEN INSERT INTO hackernews_p0067 VALUES (NEW.*);ELSIF ( NEW.id >= 16500000 AND NEW.id < 16750000)THEN INSERT INTO hackernews_p0066 VALUES (NEW.*);

ELSIF ( NEW.id >= 16250000 AND NEW.id < 16500000)THEN INSERT INTO hackernews_p0065 VALUES (NEW.*);ELSIF ( NEW.id >= 16000000 AND NEW.id < 16250000)THEN INSERT INTO hackernews_p0064 VALUES (NEW.*);ELSIF ( NEW.id >= 15750000 AND NEW.id < 16000000)THEN INSERT INTO hackernews_p0063 VALUES (NEW.*);ELSIF ( NEW.id >= 15500000 AND NEW.id < 15750000)THEN INSERT INTO hackernews_p0062 VALUES (NEW.*);ELSIF ( NEW.id >= 15250000 AND NEW.id < 15500000)THEN INSERT INTO hackernews_p0061 VALUES (NEW.*);ELSIF ( NEW.id >= 15000000 AND NEW.id < 15250000)THEN INSERT INTO hackernews_p0060 VALUES (NEW.*);ELSIF ( NEW.id >= 14750000 AND NEW.id < 15000000)THEN INSERT INTO hackernews_p0059 VALUES (NEW.*);ELSIF ( NEW.id >= 14500000 AND NEW.id < 14750000)THEN INSERT INTO hackernews_p0058 VALUES (NEW.*);ELSIF ( NEW.id >= 14250000 AND NEW.id < 14500000)THEN INSERT INTO hackernews_p0057 VALUES (NEW.*);ELSIF ( NEW.id >= 14000000 AND NEW.id < 14250000)THEN INSERT INTO hackernews_p0056 VALUES (NEW.*);ELSIF ( NEW.id >= 13750000 AND NEW.id < 14000000)THEN INSERT INTO hackernews_p0055 VALUES (NEW.*);ELSIF ( NEW.id >= 13500000 AND NEW.id < 13750000)THEN INSERT INTO hackernews_p0054 VALUES (NEW.*);ELSIF ( NEW.id >= 13250000 AND NEW.id < 13500000)THEN INSERT INTO hackernews_p0053 VALUES (NEW.*);ELSIF ( NEW.id >= 13000000 AND NEW.id < 13250000)THEN INSERT INTO hackernews_p0052 VALUES (NEW.*);ELSIF ( NEW.id >= 12750000 AND NEW.id < 13000000)THEN INSERT INTO hackernews_p0051 VALUES (NEW.*);ELSIF ( NEW.id >= 12500000 AND NEW.id < 12750000)THEN INSERT INTO hackernews_p0050 VALUES (NEW.*);ELSIF ( NEW.id >= 12250000 AND NEW.id < 12500000)THEN INSERT INTO hackernews_p0049 VALUES (NEW.*);ELSIF ( NEW.id >= 12000000 AND NEW.id < 12250000)THEN INSERT INTO hackernews_p0048 VALUES (NEW.*);ELSIF ( NEW.id >= 11750000 AND NEW.id < 12000000)THEN INSERT INTO hackernews_p0047 VALUES (NEW.*);ELSIF ( NEW.id >= 11500000 AND NEW.id < 11750000)THEN INSERT INTO hackernews_p0046 VALUES (NEW.*);ELSIF ( NEW.id >= 11250000 AND NEW.id < 11500000)THEN INSERT INTO hackernews_p0045 VALUES (NEW.*);ELSIF ( NEW.id >= 11000000 AND NEW.id < 11250000)THEN INSERT INTO hackernews_p0044 VALUES (NEW.*);ELSIF ( NEW.id >= 10750000 AND NEW.id < 11000000)THEN INSERT INTO hackernews_p0043 VALUES (NEW.*);ELSIF ( NEW.id >= 10500000 AND NEW.id < 10750000)THEN INSERT INTO hackernews_p0042 VALUES (NEW.*);ELSIF ( NEW.id >= 10250000 AND NEW.id < 10500000)THEN INSERT INTO hackernews_p0041 VALUES (NEW.*);ELSIF ( NEW.id >= 10000000 AND NEW.id < 10250000)THEN INSERT INTO hackernews_p0040 VALUES (NEW.*);ELSIF ( NEW.id >= 9750000 AND NEW.id < 10000000)THEN INSERT INTO hackernews_p0039 VALUES (NEW.*);ELSIF ( NEW.id >= 9500000 AND NEW.id < 9750000)THEN INSERT INTO hackernews_p0038 VALUES (NEW.*);ELSIF ( NEW.id >= 9250000 AND NEW.id < 9500000)THEN INSERT INTO hackernews_p0037 VALUES (NEW.*);ELSIF ( NEW.id >= 9000000 AND NEW.id < 9250000)THEN INSERT INTO hackernews_p0036 VALUES (NEW.*);ELSIF ( NEW.id >= 8750000 AND NEW.id < 9000000)THEN INSERT INTO hackernews_p0035 VALUES (NEW.*);ELSIF ( NEW.id >= 8500000 AND NEW.id < 8750000)THEN INSERT INTO hackernews_p0034 VALUES (NEW.*);ELSIF ( NEW.id >= 8250000 AND NEW.id < 8500000)THEN INSERT INTO hackernews_p0033 VALUES (NEW.*);ELSIF ( NEW.id >= 8000000 AND NEW.id < 8250000)THEN INSERT INTO hackernews_p0032 VALUES (NEW.*);

ELSIF ( NEW.id >= 7750000 AND NEW.id < 8000000)THEN INSERT INTO hackernews_p0031 VALUES (NEW.*);ELSIF ( NEW.id >= 7500000 AND NEW.id < 7750000)THEN INSERT INTO hackernews_p0030 VALUES (NEW.*);ELSIF ( NEW.id >= 7250000 AND NEW.id < 7500000)THEN INSERT INTO hackernews_p0029 VALUES (NEW.*);ELSIF ( NEW.id >= 7000000 AND NEW.id < 7250000)THEN INSERT INTO hackernews_p0028 VALUES (NEW.*);ELSIF ( NEW.id >= 6750000 AND NEW.id < 7000000)THEN INSERT INTO hackernews_p0027 VALUES (NEW.*);ELSIF ( NEW.id >= 6500000 AND NEW.id < 6750000)THEN INSERT INTO hackernews_p0026 VALUES (NEW.*);ELSIF ( NEW.id >= 6250000 AND NEW.id < 6500000)THEN INSERT INTO hackernews_p0025 VALUES (NEW.*);ELSIF ( NEW.id >= 6000000 AND NEW.id < 6250000)THEN INSERT INTO hackernews_p0024 VALUES (NEW.*);ELSIF ( NEW.id >= 5750000 AND NEW.id < 6000000)THEN INSERT INTO hackernews_p0023 VALUES (NEW.*);ELSIF ( NEW.id >= 5500000 AND NEW.id < 5750000)THEN INSERT INTO hackernews_p0022 VALUES (NEW.*);ELSIF ( NEW.id >= 5250000 AND NEW.id < 5500000)THEN INSERT INTO hackernews_p0021 VALUES (NEW.*);ELSIF ( NEW.id >= 5000000 AND NEW.id < 5250000)THEN INSERT INTO hackernews_p0020 VALUES (NEW.*);ELSIF ( NEW.id >= 4750000 AND NEW.id < 5000000)THEN INSERT INTO hackernews_p0019 VALUES (NEW.*);ELSIF ( NEW.id >= 4500000 AND NEW.id < 4750000)THEN INSERT INTO hackernews_p0018 VALUES (NEW.*);ELSIF ( NEW.id >= 4250000 AND NEW.id < 4500000)THEN INSERT INTO hackernews_p0017 VALUES (NEW.*);ELSIF ( NEW.id >= 4000000 AND NEW.id < 4250000)THEN INSERT INTO hackernews_p0016 VALUES (NEW.*);ELSIF ( NEW.id >= 3750000 AND NEW.id < 4000000)THEN INSERT INTO hackernews_p0015 VALUES (NEW.*);ELSIF ( NEW.id >= 3500000 AND NEW.id < 3750000)THEN INSERT INTO hackernews_p0014 VALUES (NEW.*);ELSIF ( NEW.id >= 3250000 AND NEW.id < 3500000)THEN INSERT INTO hackernews_p0013 VALUES (NEW.*);ELSIF ( NEW.id >= 3000000 AND NEW.id < 3250000)THEN INSERT INTO hackernews_p0012 VALUES (NEW.*);ELSIF ( NEW.id >= 2750000 AND NEW.id < 3000000)THEN INSERT INTO hackernews_p0011 VALUES (NEW.*);ELSIF ( NEW.id >= 2500000 AND NEW.id < 2750000)THEN INSERT INTO hackernews_p0010 VALUES (NEW.*);ELSIF ( NEW.id >= 2250000 AND NEW.id < 2500000)THEN INSERT INTO hackernews_p0009 VALUES (NEW.*);ELSIF ( NEW.id >= 2000000 AND NEW.id < 2250000)THEN INSERT INTO hackernews_p0008 VALUES (NEW.*);ELSIF ( NEW.id >= 1750000 AND NEW.id < 2000000)THEN INSERT INTO hackernews_p0007 VALUES (NEW.*);ELSIF ( NEW.id >= 1500000 AND NEW.id < 1750000)THEN INSERT INTO hackernews_p0006 VALUES (NEW.*);ELSIF ( NEW.id >= 1250000 AND NEW.id < 1500000)THEN INSERT INTO hackernews_p0005 VALUES (NEW.*);ELSIF ( NEW.id >= 1000000 AND NEW.id < 1250000)THEN INSERT INTO hackernews_p0004 VALUES (NEW.*);ELSIF ( NEW.id >= 750000 AND NEW.id < 1000000)THEN INSERT INTO hackernews_p0003 VALUES (NEW.*);ELSIF ( NEW.id >= 500000 AND NEW.id < 750000)THEN INSERT INTO hackernews_p0002 VALUES (NEW.*);ELSIF ( NEW.id >= 250000 AND NEW.id < 500000)THEN INSERT INTO hackernews_p0001 VALUES (NEW.*);ELSIF ( NEW.id >= 0 AND NEW.id < 250000)THEN INSERT INTO hackernews_p0000 VALUES (NEW.*);ELSE RAISE EXCEPTION 'Partition hackernews_p% does not yet exist.', NEW.id/250000;END IF;RETURN NULL;

Full example trigger

query | calls (10) | total time (10) | calls (9.6) | total time (9.6) -------+------------+-----------------+-------------+-----------------COPY | 4 | 00:03:19.7 | 4 | 01:15:11.6INSERT | (null) | (null) | 37004321 | 00:05:00.6VACUUM | 1 | 00:02:15.3 | 1 | 00:01:53CREATE | 1484 | 00:00:46.1 | 13138 | 00:00:40.8ALTER | 186 | 00:00:15.7 | 647 | 00:00:15.4

pg_stat_statements: 9.6 vs 10

query | calls (10) | total time (10) | calls (9.6) | total time (9.6) -------+------------+-----------------+-------------+-----------------COPY | 4 | 00:03:19.7 | 4 | 01:15:11.6INSERT | (null) | (null) | 37004321 | 00:05:00.6VACUUM | 1 | 00:02:15.3 | 1 | 00:01:53CREATE | 1484 | 00:00:46.1 | 13138 | 00:00:40.8ALTER | 186 | 00:00:15.7 | 647 | 00:00:15.4

pg_stat_statements: 9.6 vs 10

23 times faster

Explain

Verify your assumptions

Planner

Plan A Plan B Plan C

Explain

Explain (analyze on)

Datasets● US Environmental Protection Agency.

Air Quality System Data Marthttps://console.cloud.google.com/bigquery?p=bigquery-public-data&d=epa_historical_air_quality&page=dataset

● All stories and comments from Hacker News from its launch in 2006 to presenthttps://console.cloud.google.com/bigquery?p=bigquery-public-data&d=hacker_news&page=dataset

Partitioning deep dive

Aggregate Query

SELECT date_local, state_name, max(first_max_value) AS first_max_valueFROM epa_temperature_daily_summaryGROUP BY date_local, state_name

Aggregate Query

GroupAggregate (actual rows=431338 loops=1) Group Key: date_local, state_name Buffers: shared read=230346, temp read=35767 written=35824 I/O Timings: read=10278.527 -> Sort (actual rows=6017246 loops=1) Sort Key: date_local, state_name Sort Method: external merge Disk: 192864kB -> Seq Scan on epa_temperature_daily_summary (actual rows=6017246 loops=1)

Planning Time: 1.642 ms Execution Time: 59380.914 ms

1 single table (PostgreSQL 12)

partitioned table (PostgreSQL 12)

GroupAggregate (actual rows=431338 loops=1) Group Key: epa_temperature_daily_summary_y1990q1.date_local, epa_temperature_daily_summary_y1990q1.state_name Buffers: shared read=230406, temp read=35766 written=35823 I/O Timings: read=10640.559 -> Sort (actual rows=6017246 loops=1) Sort Key: epa_temperature_daily_summary_y1990q1.date_local, epa_temperature_daily_summary_y1990q1.state_name Sort Method: external merge Disk: 192856kB -> Append (actual rows=6017246 loops=1) -> Seq Scan on epa_temperature_daily_summary_y1990q1 (actual rows=19596 loops=1) -> Seq Scan on epa_temperature_daily_summary_y1990q2 (actual rows=20232 loops=1)

Buffers:shared read=230346temp read=35767written=35824

I/O Timings: read=10278.527

Planning Time: 1.642 msExecution Time: 59380.914 ms

Single table Partitioned

Sort and AggregatePartition

Single table

SET enable_partitionwise_aggregate TO 'on';

Enable Partitionwise Aggregate

https://www.postgresql.org/docs/current/runtime-config-query.html#GUC-ENABLE-PARTITIONWISE-AGGREGATE

Append (actual rows=431338 loops=1) Buffers: shared read=230406 I/O Timings: read=7771.143 -> HashAggregate (actual rows=2226 loops=1) Group Key: epa_temperature_daily_summary_y1990q1.date_local, epa_temperature_daily_summary_y1990q1.state_name Buffers: shared read=731 -> Seq Scan on epa_temperature_daily_summary_y1990q1 (actual rows=19596 loops=1) Buffers: shared read=731 -> HashAggregate (actual rows=2208 loops=1) Group Key: epa_temperature_daily_summary_y1990q2.date_local, epa_temperature_daily_summary_y1990q2.state_name Buffers: shared read=754 -> Seq Scan on epa_temperature_daily_summary_y1990q2 (actual rows=20232 loops=1) Buffers: shared read=754

Partitionwise Aggregate

Buffers:shared read=230406

Single table Partitionwise Agg

This is great!

Single table PartitionwisePartitioned

SELECT state_name, county_name, city_name, date_local, pressure.first_max_value, temperature.first_max_valueFROM epa_pressure_daily_summary AS pressureJOIN epa_temperature_daily_summary AS temperature

USING (date_local, state_name, county_name, city_name)

Merge Join (actual rows=3863962 loops=1) Merge Cond: ((temperature.county_name = pressure.county_name) AND (temperature.date_local = pressure.date_local) AND (temperature.state_name = pres Buffers: shared read=295773, temp read=72382 written=72470 I/O Timings: read=13941.133 -> Sort (actual rows=6017246 loops=1) Sort Key: temperature.county_name, temperature.date_local, temperature.state_name, temperature.city_name Sort Method: external merge Disk: 305376kB -> Seq Scan on epa_temperature_daily_summary temperature (actual rows=6017246 loops=1) -> Materialize (actual rows=3934054 loops=1) -> Sort (actual rows=1818026 loops=1) Sort Key: pressure.county_name, pressure.date_local, pressure.state_name, pressure.city_name Sort Method: external merge Disk: 94688kB -> Seq Scan on epa_pressure_daily_summary pressure (actual rows=1818026 loops=1)

Nonpartitioned case

Merge Join (actual rows=3863962 loops=1) Merge Cond: ((temperature.date_local = pressure.date_local) AND (temperature.state_name = pressure.state_name) AND (temperature.county_name = press Buffers: shared read=295889, temp read=72380 written=72468 I/O Timings: read=15027.763 -> Sort (actual rows=6017246 loops=1) -> Append (actual rows=6017246 loops=1) -> Seq Scan on epa_temperature_daily_summary_y1990q1 temperature (actual rows=19596 loops=1) -> Seq Scan on epa_temperature_daily_summary_y1990q2 temperature_1 (actual rows=20232 loops=1) -> Materialize (actual rows=3934054 loops=1) -> Sort (actual rows=1818026 loops=1) -> Append (actual rows=1818026 loops=1) -> Seq Scan on epa_pressure_daily_summary_y1990q1 pressure (actual rows=1513 loops=1) -> Seq Scan on epa_pressure_daily_summary_y1990q2 pressure_1 (actual rows=1490 loops=1)

Large Merge Join of 2 sorted sets

(inner) joinPartition

Single table⨝

Single table Partitioned⨝ ⨝

SET enable_partitionwise_join TO 'on';

Enable Partitionwise Join

https://www.postgresql.org/docs/current/runtime-config-query.html#GUC-ENABLE-PARTITIONWISE-JOIN

Append (actual rows=3863962 loops=1) Buffers: shared read=295889 I/O Timings: read=8642.244 -> Merge Join (actual rows=2783 loops=1) -> Sort (actual rows=1513 loops=1) -> Seq Scan on epa_pressure_daily_summary_y1990q1 pressure (actual rows=1513 loops=1) -> Sort (actual rows=19636 loops=1) -> Seq Scan on epa_temperature_daily_summary_y1990q1 temperature (actual rows=19596 loops=1) -> Merge Join (actual rows=2794 loops=1) -> Sort (actual rows=1490 loops=1) -> Seq Scan on epa_pressure_daily_summary_y1990q2 pressure_1 (actual rows=1490 loops=1) -> Sort (actual rows=20252 loops=1) -> Seq Scan on epa_temperature_daily_summary_y1990q2 temperature_1 (actual rows=20232 loops=1)

Partitionwise Join

Nonpartitioned Partitionwise Join

Single table PartitionwisePartitioned⨝ ⨝ ⨝⨝⨝⨝⨝⨝⨝⨝

Partitions need to cover the same values though

('1990-07-01') TO ('1990-10-01')('1990-07-01') TO ('1990-10-01')

⨝ ('1990-10-01') TO ('1991-01-01')('1990-10-01') TO ('1991-01-01')

⨝ ('1991-01-01') TO ('1991-04-01')('1991-01-01') TO ('1991-04-01')

⨝ ('2019-07-01') TO ('2019-10-01')('2019-07-01') TO ('2019-10-01')

Partition Pruning

During planning, during initialization, and also during execution

PREPARE hackernews_parent ASSELECT

*FROM hackernews childJOIN

hackernews parent ON (parent.id=child.parent)WHERE

child.id = $1;

Partition Pruning

Nested Loop (actual rows=1 loops=1) Buffers: shared hit=259 read=10 -> Append (actual rows=1 loops=1) -> Index Scan using hackernews_p0000_pkey on hackernews_p0000 child (actual rows=1 loops=1) -> Append (actual rows=1 loops=1) Buffers: shared hit=258 read=7 -> Index Scan using hackernews_p0000_pkey on hackernews_p0000 parent (actual rows=1 loops=1) -> Index Scan using hackernews_p0001_pkey on hackernews_p0001 parent_1 (actual rows=0 loops=1) -> Index Scan using hackernews_p0002_pkey on hackernews_p0002 parent_2 (actual rows=0 loops=1)

Planning time: 6.187 ms Execution time: 2.636 ms

Partition Pruning: PostgreSQL 10

Nested Loop (actual rows=1 loops=1) Buffers: shared hit=259 read=10 -> Append (actual rows=1 loops=1) -> Index Scan using hackernews_p0000_pkey on hackernews_p0000 child (actual rows=1 loops=1) -> Append (actual rows=1 loops=1) Buffers: shared hit=258 read=7 -> Index Scan [...] (actual rows=1 loops=1) -> Index Scan [...] (actual rows=0 loops=1) -> Index Scan [...] (actual rows=0 loops=1)

Nested Loop (actual rows=1 loops=1) Buffers: shared hit=8 -> Append (actual rows=1 loops=1) -> Index Scan using hackernews_p0000_pkey on hackernews_p0000 child (actual rows=1 loops=1) -> Append (actual rows=1 loops=1) Buffers: shared hit=4 -> Index Scan using hackernews_p0000_pkey on hackernews_p0000 parent (actual rows=1 loops=1) -> Index Scan using hackernews_p0001_pkey on hackernews_p0001 parent_1 (never executed) -> Index Scan using hackernews_p0002_pkey on hackernews_p0002 parent_2 (never executed)

Nested Loop (actual rows=1 loops=1) Buffers: shared hit=8 -> Append (actual rows=1 loops=1) -> Index Scan using hackernews_p0000_pkey on hackernews_p0000 child (actual rows=1 loops=1) -> Append (actual rows=1 loops=1) Buffers: shared hit=4 -> Index Scan [...] (actual rows=1 loops=1) -> Index Scan [...] (never executed) -> Index Scan [...] (never executed)

Buffers:shared hit=259 read=10

Planning time: 6.187 msExecution time: 2.636 ms

Buffers:shared hit=8

PostgreSQL 10 PostgreSQL 11

Summary

Summary● Declarative partitioning

Summary● Declarative partitioning● Constraints on schema design are

mostly gone (foreign keys, indexes)

mostly gone (foreign keys, indexes)● Potentially (a lot) faster

mostly gone (foreign keys, indexes)● Potentially (a lot) faster● Potentially (a lot) less server load

This is great!

Questions?

Feike Steenbergen

Software Engineer, Timescale

[email protected] · github.com/timescale

Parted ways With Partitioning? Software Engineer, Timescale · Partitioning with PostgreSQL 12...

Documents

Transcript of Parted ways With Partitioning? Software Engineer, Timescale · Partitioning with PostgreSQL 12...