greenplum.docs.pivotal.io · Contents Release Notes 3 Contents Chapter 2: Pivotal Greenplum 6.5...

PRODUCT DOCUMENTATION

Pivotal™ GreenplumDatabase®Version 6.5

Pivotal Greenplum DatabaseDocumentationRev: A01

© 2020 Pivotal Software, Inc.

Copyright Release Notes

2

Notice

Copyright

Privacy Policy | Terms of Use

Copyright © 2020 Pivotal Software, Inc. All rights reserved.

Pivotal Software, Inc. believes the information in this publication is accurate as of its publication date. Theinformation is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED"AS IS." PIVOTAL SOFTWARE, INC. ("Pivotal") MAKES NO REPRESENTATIONS OR WARRANTIES OF ANYKIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMSIMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any Pivotal software described in this publication requires an applicablesoftware license.

All trademarks used herein are the property of Pivotal or their respective owners.

Revised March 2020 (6.5.0)

http://pivotal.io/privacy-policyhttp://pivotal.io/terms-of-use

Contents Release Notes

3

Contents

Chapter 2: Pivotal Greenplum 6.5 Release Notes.................................. 14Release 6.5.0.................................................................................................................................... 15

Features.................................................................................................................................. 15Resolved Issues..................................................................................................................... 16Upgrading to Greenplum 6.5.0...............................................................................................19

Deprecated Features.........................................................................................................................20Migrating Data to Greenplum 6........................................................................................................ 22Known Issues and Limitations.......................................................................................................... 23Differences Compared to Open Source Greenplum Database.........................................................25

Chapter 3: Installing and Upgrading Greenplum................................... 26Platform Requirements......................................................................................................................27

Operating Systems................................................................................................................. 27Hardware and Network...........................................................................................................29Storage....................................................................................................................................30Tools and Extensions Compatibility....................................................................................... 30Hadoop Distributions.............................................................................................................. 32

Introduction to Greenplum.................................................................................................................34The Greenplum Master...........................................................................................................35The Segments........................................................................................................................ 35The Interconnect.....................................................................................................................39ETL Hosts for Data Loading.................................................................................................. 41Greenplum Performance Monitoring...................................................................................... 42

Estimating Storage Capacity............................................................................................................. 44Calculating Usable Disk Capacity.......................................................................................... 44Calculating User Data Size.................................................................................................... 45Calculating Space Requirements for Metadata and Logs......................................................45

Configuring Your Systems.................................................................................................................46Disabling SELinux and Firewall Software.............................................................................. 46Recommended OS Parameters Settings............................................................................... 47Synchronizing System Clocks................................................................................................ 54Creating the Greenplum Administrative User.........................................................................55Next Steps.............................................................................................................................. 56

Installing the Greenplum Database Software................................................................................... 57Installing Greenplum Database.............................................................................................. 57Enabling Passwordless SSH.................................................................................................. 58Confirming Your Installation................................................................................................... 59About Your Greenplum Database Installation........................................................................59Next Steps.............................................................................................................................. 59

Creating the Data Storage Areas......................................................................................................60Creating Data Storage Areas on the Master and Standby Master Hosts...............................60Creating Data Storage Areas on Segment Hosts.................................................................. 60Next Steps.............................................................................................................................. 61

Validating Your Systems................................................................................................................... 62Validating Network Performance............................................................................................ 62Validating Disk I/O and Memory Bandwidth...........................................................................63

Initializing a Greenplum Database System.......................................................................................64Overview................................................................................................................................. 64


4

Initializing Greenplum Database.............................................................................................64Setting Greenplum Environment Variables............................................................................ 68Next Steps.............................................................................................................................. 68

Installing Optional Extensions........................................................................................................... 70Procedural Language, Machine Learning, and Geospatial Extensions..................................70Python Data Science Module Package..................................................................................70R Data Science Library Package........................................................................................... 74Greenplum Platform Extension Framework (PXF)................................................................. 77

Installing Additional Supplied Modules..............................................................................................78Configuring Timezone and Localization Settings..............................................................................79

Configuring the Timezone...................................................................................................... 79About Locale Support in Greenplum Database..................................................................... 79Character Set Support............................................................................................................81Setting the Character Set.......................................................................................................83Character Set Conversion Between Server and Client..........................................................84

Upgrading from an Earlier Greenplum 6 Release............................................................................ 87Upgrading from 6.x to a Newer 6.x Release......................................................................... 87Troubleshooting a Failed Upgrade.........................................................................................89

Migrating Data from Greenplum 4.3 or 5..........................................................................................90Preparing the Greenplum 6 Cluster....................................................................................... 90Preparing Greenplum 4.3 and 5 Databases for Backup........................................................ 91Backing Up and Restoring a Database..................................................................................93Completing the Migration........................................................................................................94

Enabling iptables (Optional).............................................................................................................. 97Example iptables Rules..........................................................................................................97

Installation Management Utilities.....................................................................................................100Greenplum Environment Variables................................................................................................. 101

Required Environment Variables..........................................................................................101Optional Environment Variables........................................................................................... 101

Example Ansible Playbook..............................................................................................................103

Chapter 4: Greenplum Database Administrator Guide........................ 105Greenplum Database Concepts...................................................................................................... 106

About the Greenplum Architecture....................................................................................... 106About Management and Monitoring Utilities........................................................................ 109About Concurrency Control in Greenplum Database...........................................................110About Parallel Data Loading................................................................................................ 117About Redundancy and Failover in Greenplum Database...................................................118About Database Statistics in Greenplum Database............................................................. 120

Managing a Greenplum System..................................................................................................... 128About the Greenplum Database Release Version Number................................................. 128Starting and Stopping Greenplum Database....................................................................... 128Accessing the Database.......................................................................................................131Configuring the Greenplum Database System.....................................................................139Enabling Compression..........................................................................................................141Enabling High Availability and Data Consistency Features................................................. 142Backing Up and Restoring Databases................................................................................. 160Expanding a Greenplum System..........................................................................................201Migrating Data with gpcopy..................................................................................................216Monitoring a Greenplum System..........................................................................................216Routine System Maintenance Tasks....................................................................................232Recommended Monitoring and Maintenance Tasks............................................................ 236

Managing Greenplum Database Access.........................................................................................244Configuring Client Authentication......................................................................................... 244Managing Roles and Privileges............................................................................................263


5

Defining Database Objects..............................................................................................................270Creating and Managing Databases......................................................................................270Creating and Managing Tablespaces...................................................................................272Creating and Managing Schemas........................................................................................274Creating and Managing Tables............................................................................................ 276Choosing the Table Storage Model..................................................................................... 281Partitioning Large Tables......................................................................................................292Creating and Using Sequences........................................................................................... 305Using Indexes in Greenplum Database............................................................................... 308Creating and Managing Views............................................................................................. 312Creating and Managing Materialized Views.........................................................................312

Distribution and Skew..................................................................................................................... 314Local (Co-located) Joins.......................................................................................................314Data Skew............................................................................................................................ 314Processing Skew.................................................................................................................. 315

Inserting, Updating, and Deleting Data...........................................................................................318About Concurrency Control in Greenplum Database...........................................................318Inserting Rows...................................................................................................................... 319Updating Existing Rows........................................................................................................320Deleting Rows.......................................................................................................................320Working With Transactions...................................................................................................321Global Deadlock Detector.....................................................................................................322Vacuuming the Database..................................................................................................... 324Running Out of Locks...........................................................................................................324

Querying Data................................................................................................................................. 326About Greenplum Query Processing....................................................................................326About GPORCA....................................................................................................................329Defining Queries................................................................................................................... 342WITH Queries (Common Table Expressions)......................................................................354Using Functions and Operators............................................................................................358Working with JSON Data..................................................................................................... 369Working with XML Data........................................................................................................382Using Full Text Search.........................................................................................................394Using Greenplum MapReduce............................................................................................. 430Query Performance.............................................................................................................. 438Managing Spill Files Generated by Queries........................................................................ 439Query Profiling...................................................................................................................... 439

Working with External Data.............................................................................................................445Accessing External Data with PXF...................................................................................... 445Defining External Tables...................................................................................................... 445Accessing External Data with Foreign Tables..................................................................... 463Using the Greenplum Parallel File Server (gpfdist)..............................................................472

Loading and Unloading Data.......................................................................................................... 476Loading Data Using an External Table................................................................................ 477Loading and Writing Non-HDFS Custom Data.................................................................... 477Handling Load Errors............................................................................................................480Loading Data with gpload.....................................................................................................482Accessing External Data with PXF...................................................................................... 483Transforming External Data with gpfdist and gpload........................................................... 484Loading Data with COPY..................................................................................................... 494Running COPY in Single Row Error Isolation Mode............................................................495Optimizing Data Load and Query Performance................................................................... 495Unloading Data from Greenplum Database......................................................................... 495Formatting Data Files........................................................................................................... 498Example Custom Data Access Protocol.............................................................................. 501

Managing Performance................................................................................................................... 508


6

Defining Database Performance.......................................................................................... 508Common Causes of Performance Issues............................................................................ 509Greenplum Database Memory Overview............................................................................. 512Managing Resources............................................................................................................516Investigating a Performance Problem.................................................................................. 545

Chapter 5: Greenplum Database Security Configuration Guide......... 548Securing the Database....................................................................................................................549Greenplum Database Ports and Protocols..................................................................................... 550Configuring Client Authentication.................................................................................................... 554

Allowing Connections to Greenplum Database....................................................................554Editing the pg_hba.conf File.................................................................................................556Authentication Methods........................................................................................................ 557SSL Client Authentication.....................................................................................................560PAM Based Authentication...................................................................................................562Radius Authentication...........................................................................................................563Limiting Concurrent Connections......................................................................................... 563Encrypting Client/Server Connections..................................................................................564

Configuring Database Authorization................................................................................................565Access Permissions and Roles............................................................................................565Managing Object Privileges..................................................................................................565Using SSH-256 Encryption...................................................................................................566Restricting Access by Time..................................................................................................568Dropping a Time-based Restriction.................................................................................... 570

Greenplum Command Center Security........................................................................................... 571Auditing............................................................................................................................................ 574Encrypting Data and Database Connections.................................................................................. 579

Encrypting gpfdist Connections............................................................................................ 579Encrypting Data at Rest with pgcrypto.................................................................................580

Security Best Practices................................................................................................................... 588

Chapter 6: Greenplum Database Best Practices..................................592Best Practices Summary.................................................................................................................593System Configuration...................................................................................................................... 599Schema Design............................................................................................................................... 604

Data Types........................................................................................................................... 604Storage Model...................................................................................................................... 604Compression......................................................................................................................... 605Distributions.......................................................................................................................... 606Partitioning............................................................................................................................ 609Indexes..................................................................................................................................611Column Sequence and Byte Alignment............................................................................... 611

Memory and Resource Management with Resource Groups......................................................... 613Memory and Resource Management with Resource Queues........................................................ 616System Monitoring and Maintenance..............................................................................................620

Monitoring............................................................................................................................. 620Updating Statistics with ANALYZE.......................................................................................621Managing Bloat in the Database..........................................................................................622Monitoring Greenplum Database Log Files..........................................................................626

Loading Data................................................................................................................................... 628INSERT Statement with Column Values..............................................................................628COPY Statement.................................................................................................................. 628External Tables.....................................................................................................................628External Tables with Gpfdist................................................................................................ 628


7

Gpload...................................................................................................................................629Best Practices.......................................................................................................................630

Security............................................................................................................................................ 631Encrypting Data and Database Connections.................................................................................. 634Tuning SQL Queries....................................................................................................................... 643

How to Generate Explain Plans........................................................................................... 643How to Read Explain Plans................................................................................................. 643Optimizing Greenplum Queries............................................................................................ 645

High Availability............................................................................................................................... 647Disk Storage......................................................................................................................... 647Master Mirroring....................................................................................................................647Segment Mirroring................................................................................................................ 648Dual Clusters........................................................................................................................ 649Backup and Restore.............................................................................................................649Detecting Failed Master and Segment Instances................................................................ 650Segment Mirroring Configuration..........................................................................................651

Chapter 7: Greenplum Database Utility Guide..................................... 656About the Greenplum Database Utilities.........................................................................................657

Referencing IP Addresses....................................................................................................657Running Backend Server Programs.....................................................................................657

Utility Reference.............................................................................................................................. 659analyzedb..............................................................................................................................660clusterdb................................................................................................................................664createdb................................................................................................................................ 666createlang............................................................................................................................. 668createuser............................................................................................................................. 669dropdb................................................................................................................................... 672droplang................................................................................................................................ 673dropuser................................................................................................................................ 674gpactivatestandby................................................................................................................. 676gpaddmirrors......................................................................................................................... 678gpbackup_manager.............................................................................................................. 681gpbackup...............................................................................................................................685gpcheckcat............................................................................................................................ 691gpcheckperf...........................................................................................................................694gpconfig.................................................................................................................................697gpcopy...................................................................................................................................700gpdeletesystem..................................................................................................................... 700gpexpand.............................................................................................................................. 701gpfdist....................................................................................................................................705gpinitstandby......................................................................................................................... 708gpinitsystem.......................................................................................................................... 710gpload................................................................................................................................... 717gplogfilter...............................................................................................................................728gpmapreduce........................................................................................................................ 731gpmapreduce.yaml................................................................................................................732gpmovemirrors...................................................................................................................... 739gpperfmon_install..................................................................................................................741gppkg.................................................................................................................................... 745gprecoverseg........................................................................................................................ 746gpreload................................................................................................................................ 750gprestore............................................................................................................................... 752gpscp.....................................................................................................................................759gpssh.....................................................................................................................................760


8

gpssh-exkeys........................................................................................................................ 763gpstart................................................................................................................................... 765gpstate.................................................................................................................................. 767gpstop................................................................................................................................... 771gpsys1...................................................................................................................................774pg_config...............................................................................................................................775pg_dump............................................................................................................................... 777pg_dumpall............................................................................................................................785pg_restore............................................................................................................................. 790pgbouncer............................................................................................................................. 795pgbouncer.ini.........................................................................................................................796pgbouncer-admin.................................................................................................................. 808plcontainer.............................................................................................................................817plcontainer Configuration File...............................................................................................822psql........................................................................................................................................826reindexdb.............................................................................................................................. 850vacuumdb..............................................................................................................................852

Additional Supplied Programs......................................................................................................... 855

Chapter 8: Greenplum Database Reference Guide.............................. 856SQL Commands.............................................................................................................................. 857

SQL Syntax Summary..........................................................................................................860ABORT..................................................................................................................................898ALTER AGGREGATE...........................................................................................................899ALTER COLLATION.............................................................................................................900ALTER CONVERSION......................................................................................................... 901ALTER DATABASE.............................................................................................................. 902ALTER DEFAULT PRIVILEGES.......................................................................................... 903ALTER DOMAIN...................................................................................................................906ALTER EXTENSION.............................................................................................................908ALTER EXTERNAL TABLE..................................................................................................910ALTER FOREIGN DATA WRAPPER...................................................................................912ALTER FOREIGN TABLE.................................................................................................... 913ALTER FUNCTION...............................................................................................................917ALTER GROUP.................................................................................................................... 919ALTER INDEX...................................................................................................................... 920ALTER LANGUAGE............................................................................................................. 922ALTER MATERIALIZED VIEW.............................................................................................922ALTER OPERATOR............................................................................................................. 924ALTER OPERATOR CLASS................................................................................................ 925ALTER OPERATOR FAMILY...............................................................................................925ALTER PROTOCOL............................................................................................................. 928ALTER RESOURCE GROUP.............................................................................................. 929ALTER RESOURCE QUEUE...............................................................................................932ALTER ROLE....................................................................................................................... 934ALTER SCHEMA..................................................................................................................939ALTER SEQUENCE............................................................................................................. 939ALTER SERVER.................................................................................................................. 941ALTER TABLE......................................................................................................................943ALTER TABLESPACE..........................................................................................................956ALTER TEXT SEARCH CONFIGURATION.........................................................................957ALTER TEXT SEARCH DICTIONARY................................................................................ 958ALTER TEXT SEARCH PARSER........................................................................................959ALTER TEXT SEARCH TEMPLATE....................................................................................960ALTER TRIGGER.................................................................................................................960


9

ALTER TYPE........................................................................................................................961ALTER USER....................................................................................................................... 964ALTER USER MAPPING..................................................................................................... 965ALTER VIEW........................................................................................................................ 966ANALYZE..............................................................................................................................967BEGIN................................................................................................................................... 971CHECKPOINT.......................................................................................................................973CLOSE.................................................................................................................................. 974CLUSTER............................................................................................................................. 974COMMENT............................................................................................................................976COMMIT................................................................................................................................979COPY.................................................................................................................................... 980CREATE AGGREGATE........................................................................................................992CREATE CAST.....................................................................................................................998CREATE COLLATION........................................................................................................1002CREATE CONVERSION.................................................................................................... 1003CREATE DATABASE......................................................................................................... 1004CREATE DOMAIN..............................................................................................................1006CREATE EXTENSION........................................................................................................1008CREATE EXTERNAL TABLE.............................................................................................1009CREATE FOREIGN DATA WRAPPER..............................................................................1018CREATE FOREIGN TABLE............................................................................................... 1020CREATE FUNCTION..........................................................................................................1022CREATE GROUP............................................................................................................... 1031CREATE INDEX................................................................................................................. 1032CREATE LANGUAGE........................................................................................................ 1037CREATE MATERIALIZED VIEW........................................................................................1039CREATE OPERATOR........................................................................................................ 1041CREATE OPERATOR CLASS........................................................................................... 1045CREATE OPERATOR FAMILY..........................................................................................1049CREATE PROTOCOL........................................................................................................ 1050CREATE RESOURCE GROUP......................................................................................... 1051CREATE RESOURCE QUEUE..........................................................................................1054CREATE ROLE.................................................................................................................. 1057CREATE RULE...................................................................................................................1062CREATE SCHEMA.............................................................................................................1064CREATE SEQUENCE........................................................................................................ 1065CREATE SERVER............................................................................................................. 1068CREATE TABLE.................................................................................................................1069CREATE TABLE AS...........................................................................................................1083CREATE TABLESPACE.....................................................................................................1086CREATE TRIGGER............................................................................................................1088CREATE TEXT SEARCH CONFIGURATION....................................................................1090CREATE TEXT SEARCH DICTIONARY............................................................................1091CREATE TEXT SEARCH PARSER...................................................................................1092CREATE TEXT SEARCH TEMPLATE...............................................................................1093CREATE TYPE...................................................................................................................1094CREATE USER.................................................................................................................. 1102CREATE USER MAPPING................................................................................................ 1103CREATE VIEW................................................................................................................... 1103DEALLOCATE.................................................................................................................... 1106DECLARE........................................................................................................................... 1107DELETE.............................................................................................................................. 1109DISCARD............................................................................................................................ 1112DO.......................................................................................................................................1113DROP AGGREGATE..........................................................................................................1114


10

DROP CAST.......................................................................................................................1116DROP COLLATION............................................................................................................ 1116DROP CONVERSION........................................................................................................ 1117DROP DATABASE............................................................................................................. 1118DROP DOMAIN.................................................................................................................. 1118DROP EXTENSION............................................................................................................1119DROP EXTERNAL TABLE.................................................................................................1120DROP FOREIGN DATA WRAPPER..................................................................................1121DROP FOREIGN TABLE................................................................................................... 1122DROP FUNCTION..............................................................................................................1122DROP GROUP................................................................................................................... 1123DROP INDEX..................................................................................................................... 1124DROP LANGUAGE.............................................................................................................1125DROP MATERIALIZED VIEW............................................................................................1125DROP OPERATOR............................................................................................................ 1126DROP OPERATOR CLASS............................................................................................... 1127DROP OPERATOR FAMILY.............................................................................................. 1128DROP OWNED...................................................................................................................1129DROP PROTOCOL............................................................................................................ 1130DROP RESOURCE GROUP..............................................................................................1130DROP RESOURCE QUEUE.............................................................................................. 1131DROP ROLE.......................................................................................................................1132DROP RULE.......................................................................................................................1133DROP SCHEMA................................................................................................................. 1134DROP SEQUENCE............................................................................................................ 1135DROP SERVER..................................................................................................................1135DROP TABLE..................................................................................................................... 1136DROP TABLESPACE.........................................................................................................1137DROP TEXT SEARCH CONFIGURATION........................................................................1138DROP TEXT SEARCH DICTIONARY................................................................................1139DROP TEXT SEARCH PARSER....................................................................................... 1140DROP TEXT SEARCH TEMPLATE...................................................................................1140DROP TRIGGER................................................................................................................ 1141DROP TYPE....................................................................................................................... 1142DROP USER...................................................................................................................... 1143DROP USER MAPPING.....................................................................................................1143DROP VIEW....................................................................................................................... 1144END.....................................................................................................................................1144EXECUTE........................................................................................................................... 1145EXPLAIN............................................................................................................................. 1146FETCH................................................................................................................................ 1151GRANT................................................................................................................................1153INSERT............................................................................................................................... 1158LOAD.................................................................................................................................. 1161LOCK.................................................................................................................................. 1162MOVE..................................................................................................................................1165PREPARE........................................................................................................................... 1166REASSIGN OWNED...........................................................................................................1168REFRESH MATERIALIZED VIEW..................................................................................... 1169REINDEX............................................................................................................................ 1170RELEASE SAVEPOINT......................................................................................................1172RESET................................................................................................................................ 1173REVOKE............................................................................................................................. 1173ROLLBACK......................................................................................................................... 1176ROLLBACK TO SAVEPOINT.............................................................................................1177SAVEPOINT........................................................................................................................1178


11

SELECT.............................................................................................................................. 1180SELECT INTO.................................................................................................................... 1198SET..................................................................................................................................... 1199SET CONSTRAINTS.......................................................................................................... 1201SET ROLE.......................................................................................................................... 1202SET SESSION AUTHORIZATION..................................................................................... 1203SET TRANSACTION.......................................................................................................... 1204SHOW................................................................................................................................. 1207START TRANSACTION..................................................................................................... 1208TRUNCATE.........................................................................................................................1209UPDATE..............................................................................................................................1211VACUUM.............................................................................................................................1214VALUES.............................................................................................................................. 1217

Data Types.................................................................................................................................... 1220Date/Time Types................................................................................................................ 1222Pseudo-Types..................................................................................................................... 1232Text Search Data Types.................................................................................................... 1234Range Types...................................................................................................................... 1236

Summary of Built-in Functions...................................................................................................... 1241Greenplum Database Function Types................................................................................1241Built-in Functions and Operators........................................................................................1242JSON Functions and Operators......................................................................................... 1245Window Functions.............................................................................................................. 1252Advanced Aggregate Functions......................................................................................... 1254Text Search Functions and Operators............................................................................... 1256Range Functions and Operators........................................................................................ 1260

Additional Supplied Modules......................................................................................................... 1263auto_explain........................................................................................................................1263citext....................................................................................................................................1264dblink...................................................................................................................................1264diskquota.............................................................................................................................1267fuzzystrmatch...................................................................................................................... 1271gp_sparse_vector................................................................................................................1271hstore.................................................................................................................................. 1276orafce.................................................................................................................................. 1276pageinspect.........................................................................................................................1278pgcrypto.............................................................................................................................. 1278sslinfo.................................................................................................................................. 1279

Character Set Support...................................................................................................................1280Setting the Character Set...................................................................................................1282Character Set Conversion Between Server and Client...................................................... 1282

Server Configuration Parameters..................................................................................................1285Parameter Types and Values.............................................................................................1285Setting Parameters............................................................................................................. 1285Parameter Categories.........................................................................................................1286Configuration Parameters...................................................................................................1296

System Catalogs........................................................................................................................... 1377System Tables.................................................................................................................... 1377System Views..................................................................................................................... 1378System Catalogs Definitions...............................................................................................1379

The gp_toolkit Administrative Schema..........................................................................................1479Checking for Tables that Need Routine Maintenance........................................................1479Checking for Locks.............................................................................................................1480Checking Append-Optimized Tables.................................................................................. 1482Viewing Greenplum Database Server Log Files................................................................ 1486Checking Server Configuration Files..................................................................................1489


12

Checking for Failed Segments........................................................................................... 1490Checking Resource Group Activity and Status.................................................................. 1491Checking Resource Queue Activity and Status................................................................. 1495Checking Query Disk Spill Space Usage...........................................................................1497Viewing Users and Groups (Roles)....................................................................................1499Checking Database Object Sizes and Disk Space............................................................ 1499Checking for Uneven Data Distribution.............................................................................. 1503Including Data for Materialized Views................................................................................ 1504

The gpperfmon Database..............................................................................................................1506database_*.........................................................................................................................1508diskspace_*....................................................................................................................... 1509interface_stats_*................................................................................................................ 1509log_alert_*..........................................................................................................................1511queries_*............................................................................................................................. 1512segment_*..........................................................................................................................1514socket_stats_*.....................................................................................................................1515system_*............................................................................................................................. 1516dynamic_memory_info........................................................................................................ 1518memory_info...................................................................................................................... 1518

Server Programmatic Interfaces....................................................................................................1520Greenplum Partner Connector API.................................................................................... 1520Developing a Background Worker Process....................................................................... 1539

SQL Features, Reserved and Key Words, and Compliance........................................................ 1542Summary of Greenplum Features...................................................................................... 1542Reserved Identifiers and SQL Key Words......................................................................... 1552SQL 2008 Optional Feature Compliance........................................................................... 1568

Chapter 9: Greenplum Client and Loader Tools Package................. 1598

Chapter 10: About the Tools Package................................................ 1599

Chapter 11: Installing the Client and Loader Tools Package............ 1600Supported Platforms......................................................................................................................1601Installation Procedure....................................................................................................................1602About Your Installation.................................................................................................................. 1603Running the UNIX Tools Installer................................................................................................. 1604

Prerequisites....................................................................................................................... 1604Procedure............................................................................................................................1604

Running the Windows Tools Installer........................................................................................... 1605Prerequisites....................................................................................................................... 1605Procedure............................................................................................................................1605

Chapter 12: Configuring Greenplum Database for Remote ClientAccess................................................................................................. 1606

Chapter 13: Configuring a Client System for KerberosAuthentication.................................................................................... 1607

Chapter 14: Using the Client and Loader Tools................................. 1608


13

Prerequisites.................................................................................................................................. 1609Setting Up Your Greenplum Database Clients Runtime Environment.......................................... 1610Running the Client and Loader Programs.................................................................................... 1611Greenplum Database Documentation References........................................................................1612Windows Considerations............................................................................................................... 1613

Chapter 15: Client and Loader Utility Reference................................1614

Chapter 16: DataDirect ODBC Drivers for Pivotal Greenplum...........1615Prerequisites.................................................................................................................................. 1616Supported Client Platforms........................................................................................................... 1617Installing on Linux Systems.......................................................................................................... 1618

Configuring the Driver on Linux......................................................................................... 1619Testing the Driver Connection on Linux.............................................................................1620

Installing on Windows Systems.................................................................................................... 1622Verifying the Version on Windows..................................................................................... 1622Configuring and Testing the Driver on Windows................................................................1622

DataDirect Driver Documentation..................................................................................................1624

Chapter 17: DataDirect JDBC Driver for Pivotal Greenplum............. 1625Prerequisites.................................................................................................................................. 1626Downloading the DataDirect JDBC Driver.................................................................................... 1627Obtaining Version Details for the Driver....................................................................................... 1628Usage Information......................................................................................................................... 1629DataDirect Driver Documentation..................................................................................................1630

Pivotal Greenplum 6.5 Release Notes Release Notes

14

Chapter 2

Pivotal Greenplum 6.5 Release Notes

This document contains pertinent release information about Pivotal Greenplum Database 6.5 releases.For previous versions of the release notes for Greenplum Database, go to Pivotal Greenplum DatabaseDocumentation. For information about Greenplum Database end of life, see Pivotal Greenplum Databaseend of life policy.

Pivotal Greenplum 6 software is available for download from the Pivotal Greenplum page on PivotalNetwork.

Pivotal Greenplum 6 is based on the open source Greenplum Database project code.

Important: Pivotal Support does not provide support for open source versions of GreenplumDatabase. Only Pivotal Greenplum Database is supported by Pivotal Support.

https://gpdb.docs.pivotal.io/https://gpdb.docs.pivotal.io/https://support.pivotal.io/hc/en-us/articles/201143553-GPDB-software-EOL-policyhttps://support.pivotal.io/hc/en-us/articles/201143553-GPDB-software-EOL-policyhttps://network.pivotal.io/products/pivotal-gpdbhttps://network.pivotal.io/products/pivotal-gpdbhttp://greenplum.org/


15

Release 6.5.0Release Date: 2020-03-20

Pivotal Greenplum 6.5.0 is a minor release that includes changed features and resolves several issues.

Warning: The Ubuntu build of Greenplum Database 6.5.0 does not include the gpperfmondatabase, which is required for using Greenplum Command Center. Customers deploying toUbuntu should not install or upgrade to Greenplum Database 6.5 until a maintenance release isprovided to resolve this issue.

FeaturesGreenplum Database 6.5.0 includes these new and changed features:

• When creating a user-defined function, you can specify the attribute EXECUTE ON INITPLAN toindicate that the function contains an SQL command that dispatches queries to the segment instancesand requires special processing on the master instance by Greenplum Database. When possible,Greenplum Database handles the function on the master instance in the following manner:

1. First, Greenplum Database executes the function as part of an InitPlan node on the master instanceand holds the function output temporarily.

2. Then, in the MainPlan of the query plan, the function is called in an EntryDB (a special queryexecutor (QE) that runs on the master instance) and Greenplum Database returns the data that wascaptured when the function was executed as part of the InitPlan node. The function is not executedin the MainPlan.

For more information about the attribute and limitations when using the attribute, see CREATEFUNCTION.

• GPORCA introduces a new costing model for bitmap indexes. The new model is designed to choosefaster, bitmap nested loop joins instead of hash joins. The new costing model is implemented as a Betafeature, and it is used as a default only if you enable it by setting the configuration parameter:

set optimizer_cost_model = experimental

The optimizer_cost_model parameter is required only during the Beta test period for this costmodel. After further testing and validation, the new cost model will be enabled by default.

• Greenplum Database includes the server configuration parameter plan_cache_mode that controlswhether a prepared statement (either explicitly prepared or implicitly generated, for example by PL/pgSQL) can be executed using a custom plan or a generic plan plan.

Custom plans are created for each execution using its specific set of parameter values, while genericplans do not rely on the parameter values and can be re-used across executions. By default, choicebetween these options is made automatically, but it can be overridden by setting this parameter. If theprepared statement has no parameters, a generic plan is always used. The allowed values are auto(the default), force_custom_plan and force_generic_plan. This setting is considered when acached plan is to be executed, not when it is prepared.

• PXF version 5.11.1 is included, which introduces new and changed features and bug fixes. See PXFVersion 5.11.1 below.

• The s3 external table protocol automatically recognizes and uncompresses as deflate format any filethat it reads that has a .deflate extension.

• Greenplum Database introduces the Greenplum R Client Beta, an interactive in-database data analyticstool. Refer to the Greenplum Database R Client (Beta) documentation for installation and usageinformation for this tool.

The Greenplum R Client (GreenplumR) is currently a Beta feature, and is not supported for productionenvironments.


16

• gpload adds the --max_retries option to specify the number of times the utility attempts to connectto Greenplum Database after a connection timeout. The default value, 0, does not attempt a connectionafter a timeout.

• Greenplum Database introduces PL/Container version 3.0 Beta, which:

• Provides support for the new GreenplumR interface.• Reduces the number of processes created by PL/Container, in order to save system resources.• Supports more containers running concurrently.• Includes improved log messages to help diagnose problems.

PL/Container 3 is currently a Beta feature, and is not supported for production environments. It providesonly an R Docker image for executing functions; Python images are not yet available. See PL/ContainerLanguage for installation changes related to PL/Container 3.

PXF Version 5.11.1PXF includes the following new and changed features:

• PXF provides a restart command to stop, and then restart, all PXF server instances in the cluster.See Restarting PXF.

• The pxf [cluster] sync command now recognizes a [-d | --delete] option. When specified,PXF deletes files on the remote host(s) that are not present in the PXF user configuration on theGreenplum Database master host. Refer to pxf and pxf cluster.

• PXF supports filter predicate pushdown for Parquet data that you access with the Hadoop and ObjectStore Connectors. Parquet Data Type Mapping describes filter pushdown support for Parquet datatypes in PXF.

• PXF includes improvements to error handling and error surfacing.• PXF bundles newer guava and Google Cloud Storage hadoop2 libraries.• The PXF pxf-log4j.properties template file updates a log filter and changes the level from INFO

to WARN.• PXF removes unused and default Tomcat applications and files, hardening its default Tomcat security.• PXF no longer requires a $JAVA_HOME setting in gpadmin's .bashrc file on the master, standby

master, and segment hosts. You can now specify JAVA_HOME before or during PXF initialization. Referto the Initialization Overview in the PXF initialization documentation.

Resolved IssuesPivotal Greenplum 6.5.0 resolves these issues:

307 - PXF

PXF did not correctly handle an external table that was created with the ESCAPE 'OFF' orDELIMITER 'OFF' formatting options. This issue is resolved. PXF now correctly neitherescapes nor adds delimiters when reading external data with an external table created withthese options.

30155 - gpstart

On systems that use a custom tablespace or filespace, gpstart could fail to start acluster if the standby master host was down (for example, if the standby was taken offlinefor maintenance), showing the error:

Error occured while stopping the standby master: ExecutionError: 'non-zero rc: 255' occured.

This problem occurred because gpstart was attempting to check and sync the filespaceor tablespace on the unavailable standby master host. gpstart was modified to skipfilespace and tablespace checks when the standby server is not available.

30255 - Query Optimizer

../pxf/cfginitstart_pxf.html#restart_pxf../pxf/ref/pxf.html../pxf/ref/pxf-cluster.html../pxf/hdfs_parquet.html#datatype_map../pxf/init_pxf.html#init_descripthttps://github.com/greenplum-db/pxf/issues/307


17

The GPORCA cost model for bitmap indexes could would sometimes cost bitmap nestedloop joins higher than hash joins, resulting in poor query performance. GreenplumDatabase 6.5 introduces a revised cost model for bitmap indexes to address this issue.See Features.

30287 - Server: Execution

When GPORCA was enabled, queries against an append-only, column-oriented tablecould cause a PANIC due to shared memory corruption. The code was modified to guardagainst out-of-bound writes that caused the memory corruption.


For GPORCA, query performance was poor for some queries against tables with columnsthat are defined with the citext datatype. The poor performance was because GPORCAdid not gather statistics and calculate cardinalities for those columns. Now GPORCAgathers statistics and calculates cardinalities for columns defined with the citextdatatype.

30369 - Query Execution

Greenplum Database generated a PANIC when executing a query that contains a JOINLATERAL and the LATERAL subquery contains a LIMIT clause. Now the specified type ofquery completes.

30379 - ANALYZE

In some cases, performing an ANALYZE operation on a table with a column that isdefined with the citext datatype returns the error permission denied for schema. The error was generated when the user performing the operation did not haveUSAGE privilege in the schema where the citext datatype was defined with the CREATEEXTENSION citext command. Greenplum Database has been modified to not requireUSAGE privilege in the citext datatype schema for ANALYZE operations.

30382 - VACUUM,TRUNCATE

In some cases, performing a VACUUM FULL operation on the pg_class catalog table andconcurrently performing a TRUNCATE operation on a user created heap table returned theerror updated tuple is already HEAP_MOVED_OFF and caused the database to becomeunavailable. The TRUNCATE command did not properly manage the heap table entry inpg_class during the TRUNCATE operation. This issue is resolved.

30390 - gprecoverseg

In some cases, performance was poor when performing an incremental recovery withthe gprecoverseg utility on a system with a large number of segment instances.Performance is improved, now the utility performs some recovery operations in parallel.

30405 - gpcheckcat

The gpcheckcat utility failed when the dbid of Greenplum Database master was not 1.Now the master dbid is not required to be 1.


Some queries that use the window function cume_dist() return the error Backwardscanning of tuplestores are not supported if the query generates spill files.This issue is resolved and backward scanning of tuplestore spill files is allowed duringquery execution.


Queries using dynamic partition elimination (DPE) with range predicates were runningslow. This issue has been fixed by allowing only equality comparisons with DPE.

30438 - Catalog and Metadata

If the server configuration parameter gp_use_legacy_hashops was set to on,Greenplum Database incorrectly used the non-legacy opclasses when redistributing a


18

table with an ALTER TABLE...(REORGANIZE = TRUE) command if the commandcontained DISTRIBUTED BY clause the did not specify an opclass. This caused SELECTcommands against a table with redistributed data to return incorrect results.

30441 - analyzedb

The analyzedb utility could fail with an error similar to ERROR: relation"pg_aoseg.pg_aocsseg_xxxxxx" does not exist if a table was dropped duringthe analyzedb operation. This problem was resolved by ensuring that analyzedb skipsany dropped tables when determining the list of tables to analyze.

30450 - PXF

PXF initialization and reset failed when the default system Java version differed from thatspecified in PXF's $JAVA_HOME. This issue is resolved; PXF has added flexibility to thespecification of the $JAVA_HOME setting.

30452 - Dispatch

If the server configuration parameter check_function_bodies was set in a session onthe master, the parameter setting did not persist when a related segment instance sessionwas reset. This caused some functions to fail. Now the parameter setting persists when asegment instance session is reset.


GPORCA incorrectly determined that the plan for a query with a filter on a window functionover the distribution key column was direct dispatchable. Now direct dispatch requires thefilter to be on a table scan.

30471, 8987 - Postgres Planner

When the Postgres Planner executed some queries that contain a subquery that containboth a distinct qualified aggregate expression and a GROUP BY clause, the PostgresPlanner returned the error could not find pathkey item to sort. The error wasreturned when the Postgres Planner supply did not properly manage information used forsorting.


In some specific situations, some specific types of queries generated a GreenplumDatabase PANIC. The PANIC occurred when Greenplum Database did not properlyhandle skew optimization for multi-batch hash joins. Criteria for a query that caused thePANIC include the query contains a join, the join key has segment-local statistics (such asa catalog table), and the join key is one of the most common values, and the query plan ismulti-batch hash join and the hash join is rescannable.

30477 - Query Analyze

While gathering statistics for a partitioned table, the pg_class columns relpages andreltuples were not populated for the root partition, only for leaf partitions. This issue hasbeen fixed by changing the method to calculate if a partition table is empty or not.

30485 - gpinitsystem

When initializing a Greenplum Database system, the gpinitsystem utility failed to setthe password for a user name when the name is numeric.

30493 - analyzedb

When used with the --config-file option, analyzedb did not enumerate the leafpartitions of a partitioned table and processed the root partition as a non-partitionedtable. For heap tables this produced an error. For append-optimized tables, no error wasraised, but DML changes to leaf partitions were not tracked properly. This issue has beenresolved. Using the --config-file option correctly analyzes partitioned tables.

168199193 - COPY

https://github.com/greenplum-db/gpdb/issues/8987


19

In some cases, performance of the COPY command in Greenplum Database 6 was poorwhen compared to Greenplum Database 5. The performance of the COPY command isimproved.

168828451, 8677 - Planner

Some queries returned incorrect results when the queries contain subqueries that performa join and also contain one or more equality predicates, and optionally contain an ISNULL predicate. Incorrect results were returned when either a merge join or a nested loopjoin did not correctly process the predicates. This issue is resolved.

169030090 - Server

Superusers were limited to 3 connections by default, causing "too many clients" errorswhen users run maintenance scripts. The maximum number of superuser connections isset with the superuser_default_connections server configuration parameter. Thisissue is resolved. The default value for this parameter has changed from 3 to 10.

170745356, 9407 - Query Execution

With gp_enable_global_deadlock_detector set to on, concurrent updates to thesame table could produce an incorrect query result. This issue is resolved. Segmentsreport waited-for transaction IDs to the master so that the master has the same transactionorder as the segments.

170861600 - Server

Using ALTER TABLE tablename SPLIT PARTITION could cause rows to be assignedto the wrong partition, or could cause a crash, if one or more columns before the partitionkey were dropped. This issue is resolved.

171481916 - gpinitstandby

In some cases, utilities that checked for host IP address such as gpinitstandby failed.A Python utility (ifaddrs) that is used those Greenplum Database utilities caused thefailure. ifaddrs has been updated.

171596248 9679 Query Execution

A Greenplum Database segment instance might generate a PANIC when a query that joinstables with a compound data type generates a query plan that performs a data motion andcontains Nested Loop joins. The PANIC occurs due to an error in the prefetch logic for themotion. The prefetch logic issue has been corrected.

Upgrading to Greenplum 6.5.0Note: Greenplum 6 does not support direct upgrades from Greenplum 4 or Greenplum 5 releases,or from earlier Greenplum 6 Beta releases.

See Upgrading from an Earlier Greenplum 6 Release to upgrade your existing Greenplum 6.x software toGreenplum 6.5.0.

https://github.com/greenplum-db/gpdb/issues/8677https://github.com/greenplum-db/gpdb/issues/9407https://github.com/greenplum-db/gpdb/issues/9679


20

Deprecated FeaturesDeprecated features will be removed in a future major release of Greenplum Database. Pivotal Greenplum6.x deprecates:

• The analzyedb option --skip_root_stats (deprecated since 6.2).

If the option is specified, a warning is issued stating that the option will be ignored.• The server configuration parameter gp_statistics_use_fkeys (deprecated since 6.2).• The follo

greenplum.docs.pivotal.io · Contents Release Notes 3 Contents Chapter 2: Pivotal Greenplum 6.5...

Documents

Transcript of greenplum.docs.pivotal.io · Contents Release Notes 3 Contents Chapter 2: Pivotal Greenplum 6.5...