Biml Academy 2 - Lesson 5: Importing source metadata into Biml

Biml Academy 2

Importing, Staging, and Loading Data with Biml

Andy Leonard · Scott Currie · Cathrine Wilhelmsen

Biml Academy 2

Lesson 5: Importing source metadata into Biml (Cathrine)

Lesson 6: Staging data with Biml (Andy)

Lesson 7: Metadata-driven staging pattern with Biml (Andy)

Lesson 8: BimlFlex for Data Vault data warehouses (Scott)

Lesson 9: Biml Q&A with several BimlHeroes!

Cathrine Wilhelmsen

@cathrinew

cathrinew.net

Data Warehouse ArchitectBusiness Intelligence Developer

Lesson 5:

Importing source metadata into Biml

Methods for importing metadata

ImportTableNodes

ImportDB

GetDatabaseSchema


ImportTableNodes

Import tables and views in one schema

Filter tables using LIKE % wildcard syntax


ImportDB

Import all schemas, tables and views

Filter using LIKE % wildcard syntax


GetDatabaseSchema

Import all schemas, tables and views

Filter using collections

Why ImportDB and GetDatabaseSchema?

ImportDB / ImportTableNodes

• SELECT * on each imported table

• SELECT to get defaults, indexes etc.

• Using SchemaOnly mode

• Doesn't actually run the query, but returns schema for the result set

GetDatabaseSchema

• Figures out target database type (SQL Server, Oracle, Teradata, DB2, PostgreSQL, Ingres, OpenEdge etc.)

• Queries the relevant information schema for that database type

A few queries for the entire importMultiple queries per table

But why keep both if one is faster?

Support for legacy scripts :)

They can return different results – where both are valid/correct

Different data type mapping defaults in different systems(Bulk Copy, SSIS and Excel are all different)

Biml supports the two most commonly used approaches

Methods and parameters

ImportTableNodes(schema, tableFilter)

ImportTableNodes(schema, tableFilter, importOptions)

ImportDB()

ImportDB(schemaFilter, tableFilter)

ImportDB(schemaFilter, tableFilter, importOptions)

GetDatabaseSchema()

GetDatabaseSchema(importOptions)

GetDatabaseSchema(includedSchemas, includedTables, importOptions)

Methods and parameters: string value



ImportDB()



GetDatabaseSchema()



Methods and parameters: string filter



ImportDB()



GetDatabaseSchema()



Methods and parameters: collections



ImportDB()



GetDatabaseSchema()



Methods and parameters: ImportOptions



ImportDB()



GetDatabaseSchema()



ImportOptions specify objects to exclude

Don't exclude anything

Views

Identity Specifications

Primary Keys

Foreign Keys

Unique Keys

Indexes

Column Defaults

Check Constraints

ImportOptions.None

ImportOptions.ExcludeViews

ImportOptions.ExcludeIdentity

ImportOptions.ExcludePrimaryKey

ImportOptions.ExcludeForeignKey

ImportOptions.ExcludeUniqueKey

ImportOptions.ExcludeIndex

ImportOptions.ExcludeColumnDefault

ImportOptions.ExcludeCheckConstraint

Demo time!

Additional methods for importing metadata

GenerateSchemaNode

Returns AstSchemaNode

GenerateSchemaNodes

Returns IEnumerable<AstSchemaNode>

Additional methods for importing metadata

GenerateTableNode

Returns AstTableNode

GenerateTableNodes

Returns IEnumerable<AstTableNode>

Demo time!

Biml Academy 2

Lesson 5: Importing source metadata into Biml (Cathrine)

Lesson 6: Staging data with Biml (Andy)

Lesson 7: Metadata-driven staging pattern with Biml (Andy)

Lesson 8: BimlFlex for delivering Data Vault data warehouses (Scott)

Lesson 9: Biml Q&A with several BimlHeroes!

Biml Academy 2 - Lesson 5: Importing source metadata into Biml

Data & Analytics

Transcript of Biml Academy 2 - Lesson 5: Importing source metadata into Biml