Gebruikersdag 2018

Power BI Custom ConnectorsJeroen ter Heerdt

jeroen.ter.heerdt@microsoft.com@jeroenterheerdthttp://www.dutchdatadude.com

Power Query

Power Query in Excel

Power Query in Power BI Desktop

Power Query in SQL Server Data Tools

Power Query Editor

Power QueryDesign Tenents

Design Tenets for “M”

Power QuerySyntax and language flow

Syntax for a Simple Query

SELECT Orders.OrderDate, Products.OrderID, Products.ProductSKU

FROM Products

INNER JOIN Orders ON Products.OrderID = Orders.OrderID

ORDER BY Products.ProductSKU

from p in Products

join o in Orders on p.OrderID equals o.OrderID

orderby p.ProductSKU

select new { o.OrderDate, p.OrderID, p.ProductSKU }

Products.Join(Orders, Products,

o => o.OrderID, p => p.OrderID,

(p, o) => new { o.OrderDate, p.OrderID, p.ProductSKU }

).OrderBy( p => p.ProductSKU )

let

Joined = Table.Join( Products, "OrderID", Orders, "OrderID" ),

Columns = Table.SelectColumns(Joined, {"OrderDate", "OrderID", "ProductSKU"}),

Sorted = Table.Sort( Columns, "ProductSKU" ),

in

Sorted

Language Flow

lazy evaluation

letSource = Web.Page(Web.Contents("http://www.bing.com/blogs/site_blogs/b/search/archive/2013/12/01/eoy.aspx")),WebTable = Source{index}[Data],RenamedColumns = Table.RenameColumns(WebTable,{{"Column1", "Rank"}, {"Column2", "2013"}})

inRenamedColumns

letSource = Sql.Database("localhost", "AdventureWorksDW2012"),DimCat = Source{[Schema="dbo",Item="DimProductCategory"]}[Data],DimSubCat = Source{[Schema="dbo",Item="DimProductSubcategory"]}[Data],DimCustomer = Source{[Schema="dbo",Item="DimCustomer"]}[Data],Joined = Table.NestedJoin(DimSubCat,{"CategoryKey"},DimCat,{"CategoryKey"},"Category",JoinKind.Inner)

inJoined

How it Actually Works

letSource = Sql.Database("localhost", "AdventureWorksDW2012"),DimCat = Source{[Schema="dbo",Item="DimProductCategory"]}[Data],DimSubCat = Source{[Schema="dbo",Item="DimProductSubcategory"]}[Data],DimCustomer = Source{[Schema="dbo",Item="DimCustomer"]}[Data],Joined = Table.NestedJoin(DimSubCat,{"CategoryKey"},DimCat,{"CategoryKey"},"Category",JoinKind.Inner)

inJoined

Joined

DimSubCat

DimCat

Source

Source

Values are lazy – they are only evaluated when they are “pulled”

Power QueryQuery Folding

Query Folding (Delegation, Pushdown)

let

Source = Sql.Database("localhost", "AdventureWorksDW2012"),

Products = Source{[Schema="dbo",Item="DimProduct"]}[Data],

RemovedOtherColumns = Table.SelectColumns(Products,{"ProductKey", "Name"}),

RenamedColumns = Table.RenameColumns(RemovedOtherColumns,{{"Name", "Product"}}),

FilteredRows = Table.SelectRows(RenamedColumns, each [ProductKey] < 10),

Sorted = Table.Sort(FilteredRows,{{"ProductKey", Order.Ascending}})

in

Sorted

Mashup Engine

Folding to Query Languages like SQL

SELECT ProductKey, Name as [Product]FROM [dbo].[DimProduct]WHERE [ProductKey] < 10ORDER BY [ProductKey]

let

Source = Sql.Database("localhost", "AdventureWorksDW2012"),

Products = Source{[Schema="dbo",Item="DimProduct"]}[Data],

RemovedOtherColumns = Table.SelectColumns(Products,{"ProductKey", "Name"}),

RenamedColumns = Table.RenameColumns(RemovedOtherColumns,{{"Name", "Product"}}),

FilteredRows = Table.SelectRows(RenamedColumns, each [ProductKey] < 10),

Sorted = Table.Sort(FilteredRows,{{"ProductKey", Order.Ascending}})

in

Sorted

The Mashup Engine generates

queries in the native language of

the source, pushing down as

much work as possible to the

backend. Any remaining work is

performed locally.

Join's across sources can result in

filters or join logic being passed

from one side to the other.

Partial Folding and Compensation

Mashup Engine

Query

Query

https://sharepoint.com/.../?filter=ProductKey gt 10

File is read and processed locally

Filters pushed down as query string parameters

Rename and sort

done locally

Rename, filter,

and sort done

locally

What Gets Folded?

Power QueryCredentials and Privacy Levels

Credentials and Privacy Levels

Power QueryType system

Symbolic Operators

• Arithmetic (numbers, durations, and date-time values) +x -x x + y x - y x * y x / y

• Equality, inequality x = y x <> y

• Comparison x < y x <= y x > y x >= y

• Concatenation (text & text, list & list, record & record, table & table, date & time)

x & y

• Field access (or null) r[f] r[f]?

• Projection (fill with null) r[[f1],[f2]] r[[f1],[f2]]?

• Item lookup (or null) l{i} l{i}?

• Item lookup by key (or null) l{[k1=v1, k2=v2]} l{[k1=v1, k2=v2]}?

• Function application f(a1, a2)

• Not-implemented error …

• Self-recursive reference @n

Type System

A small set of built-in types• any, none

• null, logical, number, text, binary

• time, date, datetime, datetimezone, duration

Complex Types• list, record, table, function

Sql.Database = (server as text, database as text, optional options as nullable record) as table

MyCoolFunction = (index as number, category as text) as nullable table

Date.StartOfDay = (dateTime as any) as any

Ascribed Types

A value's ascribed type is the type to which a value is declared to conform. When a

value is ascribed a type, only a limited conformance check occurs. M does not perform

conformance checking beyond a nullable primitive type. M program authors that

choose to ascribe values with type definitions more complex than a nullable primitive-

type must ensure that such values conform to these types.

Commonly used ascribed types• Byte.Type, Int8.Type, Int16.Type, Int32.Type, Int64.Type

• Single.Type, Double.Type, Decimal.Type, Currency.Type, Percentage.Type

• Character.Type

Simple Values

SIMPLE VALUE LITERAL

Null null

Logical true, false

Number 1, 1.2, 1.2e-3, #infinity, #nan

Text "hello, world!"

Date #date(2013, 3, 8)

Time #time(15, 10, 0)

DateTime #datetime(2013, 3, 8, 15, 10, 0)

DateTimeZone #datetimezone(2013, 3, 8, 15, 10, 0, -8, 0)

Duration #duration(1, 13, 59, 12.34)

Complex Values - Lists

Complex Values - Records

Complex Values – Tables

Complex Values - Others

COMPLEX VALUE LITERAL

Function MyFunction = ( x, y, optional z ) => if z = null then

x + y else

(x + y) / z

Type type table [ n = number, #"n^2" = number ]

Binary #binary({0x68, 0x65, 0x6C, 0x6C, 0x6F})

Anatomy of a Table

letSource = Excel.Workbook(File.Contents("Sales.xlsx")),MyTable = Source{[Item="Data",Kind="Sheet"]}[Data]

inMyTable

Anatomy of a Table

MyTable[ProductKey] = { 217, 231, 485, 538, 480, 528, 480, 477 }

A column in the table is a list.

Anatomy of a Table

A row in the table is a record.

MyTable{2} = [ ProductKey = 485, OrderQuantity = 1, UnitPrice = 21.98,SalesAmount = 21.98, OrderDate = #date(2008, 1, 1) ]

Anatomy of a Table

A table is a list of records

MyTable = {[ProductKey = 217,…],[ProductKey = 231,…],…}

Unary functions

• Many library functions take functions as arguments

• Often, those parameter functions are unary• A special syntactic form helps construct unary function values

• An ‘each’ expression is just shorthand for a unary function• The single parameter of an ‘each’ function is named _

• For conciseness and to get close to the DAX syntax, the _ can be omitted when accessing fields or columns

Table.SelectRows( table, (r) => r[Manager] = r[Buddy] )

Table.SelectRows( table, each _[Manager] = _[Buddy] )

Table.SelectRows( table, each [Manager] = [Buddy] )

Special Syntactic Forms

Conditional expression

Let expression

Error expression (think: “throw”)Terminates evaluation and raises error

Try expression (think: “catch”)Attempts evaluation, optional “otherwise”

Encodes results and errors as record values

if 1 < 2 then "hurray" else "sad face"

let x = Number.Atan(3) in x * x

error "Here I go wrong"error [ Reason = "Expression.Error",

Message = "'t' should be positive", Detail = t ]

try error "Bad"// [ HasError = true,// Error = [// Reason = "Expression.Error",// Message = "Bad",// Detail = null// ]// ]try error "Bad" otherwise 42// 42

Power QueryWriting M and tips and tricks

Writing M

Advanced ViewAccess/edit the full query text in the Advanced View

The formula for the current step is shown in the formula bar

The “fx” button creates a new step, referencing the previous step by name

Case Sensitive!Functions are typically CamelCase

Library functions are usually named in the form of Type.Action

Table.SelectColumns, Number.ToText, etc.

Special syntactic forms are usually lower caseif, then, etc.

Tips and Tricks: Leverage the UI

Generate steps using the UI, then tweak the code

Some things only work through the UI (“auto steps”)

Tips and Tricks: Library Functions

Use #shared to see all exported functions (and keywords)

Typing in the function name will display its help and prompt for parameter

Tips and Tricks: Troubleshooting

Use try/catch to isolate errors

Select and remove rows with errors

Table.Buffer will stop folding from occurring

Mashup EngineExtensibility – teaching new tricks

M Extensibility

Allows definition of new functions

• Dynamically loaded at runtime

• Can be used to add new Data Source functions

•Associated with a “Data Source Kind” and credential

Extensions packaged as a zip file (.mez, or .pqx)

• Contains M definition, icons, and string resources

M Extensibility: how it works

Data Connector Extensibility

• Extend Power BI data source

options through OData/ODBC/M

• Detects service capabilities

• Sandboxed, cloud deployable

• M Extension is a lightweight

wrapper providing:

• Icons and branding

• Compensation and query

generation

• Custom authentication flows

Data

Source

Simple OData Examplesection TripPin;

// This record declares that TripPin.Feed should be recognized as data source function of kind "TripPin". The Publish record// indicates that it should be shown in the Power Query "Get Data” dialog.[DataSource.Kind="TripPin", Publish="TripPin.Publish"]shared TripPin.Feed = (url as text) =>

let// Wrapping another data source function gives us all the capabilities of that function// (Paging, OData $metadata, query folding).source = OData.Feed(url)

insource;

// Data Source Kind descriptionTripPin = [

// Declares the supported type(s) of authentication. In this case, Implicit = Anonymous web access.Authentication = [

Implicit = []],// Assigns a label to the data source credential. This will be displayed in the "Manage Data Sources" dialog.Label = "TripPin Part 1 - OData"

];

// Data Source UI publishing descriptionTripPin.Publish = [

Beta = true,Category = "Other",ButtonText = { "TripPin OData", "TripPin OData" }

];

Mashup EngineExtensibility – demo

Resources

https://github.com/Microsoft/DataConnectors

Gebruikersdag 2018

Bedankt!Vergeet niet de evaluatie in te vullen!Scan de QR code.