Unified Logging and Activity Tracing

© 2016 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.

Logging for the future

System Frameworks #WWDC16

Session 721

Unified Logging and Activity Tracing

Steven Szymanski Core OS EngineeringMatthieu Lucas System Applications

Agenda

IntroductionLogging ConceptsDemoUsing the Unified SystemToolsBest PracticesGathering LogsDeprications

Introduction

Background

Background

In 2014 Apple introduced Activity Tracing

Background

In 2014 Apple introduced Activity TracingWe also introduced the concept of Faults and Errors

Background

In 2014 Apple introduced Activity TracingWe also introduced the concept of Faults and ErrorsWe recognize that Apple has several logging APIs

Goals

One common, efficient logging mechanism for both user and kernel mode

Goals

One common, efficient logging mechanism for both user and kernel modeMaximize information collected while minimizing observer effect

Goals

One common, efficient logging mechanism for both user and kernel modeMaximize information collected while minimizing observer effect• Compressing data

Goals

One common, efficient logging mechanism for both user and kernel modeMaximize information collected while minimizing observer effect• Compressing data• Deferring work and data collection

Goals

One common, efficient logging mechanism for both user and kernel modeMaximize information collected while minimizing observer effect• Compressing data• Deferring work and data collection• Managing log message lifecycle

Goals


We want as much logging on all the time as possible

Goals


We want as much logging on all the time as possibleDesign privacy into the system

Features

Features

Improved categorization and filtering of log messages

Features

Improved categorization and filtering of log messagesLogging system collects caller information for you

Features

Improved categorization and filtering of log messagesLogging system collects caller information for youNew builtin type specifiers - simplifies log message preparation

Features

Improved categorization and filtering of log messagesLogging system collects caller information for youNew builtin type specifiers - simplifies log message preparationNew Console application and command-line tool

Features

Improved categorization and filtering of log messagesLogging system collects caller information for youNew builtin type specifiers - simplifies log message preparationNew Console application and command-line toolSupported on macOS, iOS, tvOS, watchOS, and Simulators

Features

Improved categorization and filtering of log messagesLogging system collects caller information for youNew builtin type specifiers - simplifies log message preparationNew Console application and command-line toolSupported on macOS, iOS, tvOS, watchOS, and SimulatorsSupport for Objective-C, C++ and C

Features

Improved categorization and filtering of log messagesLogging system collects caller information for youNew builtin type specifiers - simplifies log message preparationNew Console application and command-line toolSupported on macOS, iOS, tvOS, watchOS, and SimulatorsSupport for Objective-C, C++ and C Swift support in upcoming seed

Current Console

Console Revisited…

Logging Concepts

Adoption

Adoption

If you want to use the new Unified Logging system• Build with the macOS 10.12, iOS 10.0, tvOS 10.0 or watchOS 3.0 SDK• Legacy APIs (NSLog, asl_log_message, syslog…) redirected into new system• Log data will be in new format and location

Adoption

If you want to use the new Unified Logging system• Build with the macOS 10.12, iOS 10.0, tvOS 10.0 or watchOS 3.0 SDK• Legacy APIs (NSLog, asl_log_message, syslog…) redirected into new system• Log data will be in new format and location

If you don’t want to use the new Unified Logging system• Build with macOS10.11, iOS 9.0, tvOS 9.0 and watchOS 2.0 SDK• No changes

New File Formats

New File Formats

Log data is kept in a compressed binary format: .tracev3 files

New File Formats

Log data is kept in a compressed binary format: .tracev3 filesStored under /var/db/diagnostics/ with support in /var/db/uuidtext

New File Formats

Log data is kept in a compressed binary format: .tracev3 filesStored under /var/db/diagnostics/ with support in /var/db/uuidtextNew tools to access the stored and live log messages

New File Formats

Log data is kept in a compressed binary format: .tracev3 filesStored under /var/db/diagnostics/ with support in /var/db/uuidtextNew tools to access the stored and live log messages• Because data is stored in binary format, you MUST use new tools to access files

New File Formats

Log data is kept in a compressed binary format: .tracev3 filesStored under /var/db/diagnostics/ with support in /var/db/uuidtextNew tools to access the stored and live log messages• Because data is stored in binary format, you MUST use new tools to access files

New .logarchive format for portability of logs

NEWSubsystems and Categories


Log messages can be associated with a subsystem and category


Log messages can be associated with a subsystem and categoryCan be used to control how log messages are filtered and displayed


Log messages can be associated with a subsystem and categoryCan be used to control how log messages are filtered and displayedA subsystem can contain multiple categories


Log messages can be associated with a subsystem and categoryCan be used to control how log messages are filtered and displayedA subsystem can contain multiple categoriesYou can use as many subsystems and categories as needed



Example:



Subsystem Category

com.your-company.your-application setup, inprogress, teardown

Example:



Subsystem Category

com.your-company.your-application setup, inprogress, teardown

com.your-company.test.your-application test

Example:

Logging Behavior NEW

Logging Behavior

Each log message has a level determined by the API used

NEW

Logging Behavior

Each log message has a level determined by the API used• Three basic levels—Default, Info, Debug

NEW

Logging Behavior

Each log message has a level determined by the API used• Three basic levels—Default, Info, Debug• Two special levels—Fault, Error

NEW

Logging Behavior


Each basic level has two characteristics that can be set for system, subsystem, or category

NEW

Logging Behavior


Each basic level has two characteristics that can be set for system, subsystem, or category• Is is enabled? (Default messages are always enabled)

NEW

Logging Behavior


Each basic level has two characteristics that can be set for system, subsystem, or category• Is is enabled? (Default messages are always enabled)• Is it stored to disk or memory?

NEW

Logging Behavior



The levels are hierarchical

NEW

Logging Behavior



The levels are hierarchical• So setting Debug to go to disk implies that Info will also go to disk

NEW

Logging Behavior



The levels are hierarchical• So setting Debug to go to disk implies that Info will also go to disk

Behavior can be customized by installing profiles or, on macOS, via log command

NEW

Standard Behavior NEW

Standard Behavior

Message Level Enabled Destination

DEFAULT LEVEL ALWAYS DISK

INFO LEVEL YES MEMORY

DEBUG LEVEL NO N/A

NEW

Standard Behavior

Message Level Enabled Destination

DEFAULT LEVEL ALWAYS DISK

INFO LEVEL YES MEMORY

DEBUG LEVEL NO N/A

ERROR ALWAYS DISK

FAULT ALWAYS DISK

NEW

Privacy NEW

Privacy

Prevent accidental logging of Personally Identifiable Information (PII)

NEW

Privacy

Prevent accidental logging of Personally Identifiable Information (PII)Dynamic strings, collections, arrays, etc. are assumed to be private

NEW

Faults and Errors NEW

Faults and Errors

We do extra work saving additional information on Fault or Error

NEW

Faults and Errors

We do extra work saving additional information on Fault or Error Errors represent issues discovered within a given application/library

NEW

Faults and Errors

We do extra work saving additional information on Fault or Error Errors represent issues discovered within a given application/libraryFaults represent more global problems in the system

NEW

Faults and Errors

We do extra work saving additional information on Fault or Error Errors represent issues discovered within a given application/libraryFaults represent more global problems in the systemFaults and Error log information is captured into a separate set of log files

NEW

Architecture

Architecture

Process A

Process B

4 k4 kBuffer

4 k4 kBuffer

Architecture

Process A logd

Compressor

Process B

4 k4 kBuffer

BufferBufferBufferBufferCompressed Buffer

4 k4 kBuffer

Architecture

Process A logd

Compressor

Process B

4 k4 kBufferMemory Only Buffer

Regular Log Data

Fault and Error Log Data

Others

Compressed Log Files


4 k4 kBuffer

Architecture

Process A logd

Compressor

Process B


Regular Log Data


Others“Live” log stream

Compressed Log Filesdiagnosticd


4 k4 kBuffer

Architecture

Process A logd

Compressor

Process B

Profile can change routing and rules for given applications or subsystems


Regular Log Data


Others“Live” log stream

Compressed Log Filesdiagnosticd


4 k4 kBuffer

DemoConsole demo

Using the Unified System

API Destination Description

os_log Disk Default logging level that is always captured

os_log_info Memory Additional information (defaults to memory-only buffers)

os_log_debug Off Debug level content (off-by default)

os_log_error Disk Process local error

os_log_fault Disk System-level error (usually involves multiple processes)

os_log_create n/a Create a log object for custom behaviors

Summary of New APIs NEW

Creating a Log Object NEW


os_log_t log = os_log_create("com.your_company.subsystem", "network");

Create thread-safe singleton object that controls behavior of log messages



Create thread-safe singleton object that controls behavior of log messages Defaults to system-behavior



Create thread-safe singleton object that controls behavior of log messages Defaults to system-behaviorUsage

os_log(log, "This happened");





Reference to category and subsystem stored with every log message





Reference to category and subsystem stored with every log messageOr use OS_LOG_DEFAULT • For messages not associated with subsystem/category

Built-in Type Formatters NEW

We all spend too much code converting binary information to strings to log



Built-in decoding for common values"%{time_t}d" or "%{errno}d"



Built-in decoding for common values"%{time_t}d" or "%{errno}d"Arbitrary binary data using a new format type"%.*P"



Built-in decoding for common values"%{time_t}d" or "%{errno}d"Arbitrary binary data using a new format type"%.*P"

Built-in decoding for binary-types“%{uuid_t}.16P"

Example Type Formatters

Type Format String Example Outputtime_t %{time_t}d 2016-01-12 19:41:37

timeval %{timeval}.*P 2016-01-12 19:41:37.774236

timespec %{timespec}.*P 2016-01-12 19:41:37.774236823

errno %{errno}d Broken pipe

uuid_t %{uuid_t}.16P %{uuid_t}.*P 10742E39-0657-41F8-AB99-878C5EC2DCAA

sockaddr %{network:sockaddr}.*P fe80::f:86ff:fee9:5c16 17.43.23.87

in_addr %{network:in_addr}d 17.43.23.87

in6_addr %{network:in6_addr}.16P fe80::f:86ff:fee9:5c16

NEW

Per Parameter Privacy NEW

Per Parameter Privacy

Privacy is handled on a parameter by parameter basis

NEW


Privacy is handled on a parameter by parameter basisScalars and static strings are assumed to be public

NEW


Privacy is handled on a parameter by parameter basisScalars and static strings are assumed to be publicDynamic strings, collections, and objects are assumed to be private

NEW



NEW

Can be overridden on a per-parameter basis

"%{public}@" or "%{private}d"



NEW

Can be overridden on a per-parameter basis

"%{public}@" or "%{private}d"Combine privacy and formatting

"%{public, uuid_t}.16P”

/*

* Log Message Simplification

*/

/*


*/

// Old way:

/*


*/

// Old way:

if (LogLevelEnabled(info)) {

/*


*/

// Old way:


uuid_string_t uuid_str;

uuid_unparse_upper(uuid, uuid_str);

/*


*/

// Old way:




char *addr_desc = _convert_sockaddr(&sa);

/*


*/

// Old way:





NSLog(@"%s (%s:%d) - fd: %d, uuid: %s, IP: %s”,

__PRETTY_FUNCTION__, __FILE__, __LINE__, fd, uuid_st, addr_desc);

/*


*/

// Old way:







free(addr_desc);

/*


*/

// Old way:







free(addr_desc);

}

/*


*/

// Old way:







free(addr_desc);

}

// New way:

/*


*/

// Old way:







free(addr_desc);

}

// New way:

os_log_info(OS_LOG_DEFAULT,

/*


*/

// Old way:







free(addr_desc);

}

// New way:


"fd: %d, uuid: %{uuid_t}.16P, IP: %{network:sockaddr}.*P”,

/*


*/

// Old way:







free(addr_desc);

}

// New way:


"fd: %d, uuid: %{uuid_t}.16P, IP: %{network:sockaddr}.*P”,

fd, uuid, sa->sa_len, &sa);

/* * Example Code */


os_log_t general_log = os_log_create("com.apple.logging.example", "general");


os_log_t general_log = os_log_create("com.apple.logging.example", "general");os_log_t time_log = os_log_create("com.apple.logging.example", "timestamp");



os_log(general_log, "running example code”);




os_log_info(general_log, "processing file %{public}s”, filename);





int fd = open(filename, O_RDONLY);if (fd < 0) { os_log_error(general_log, "Cannot open file %{public}s - %{errno}d", filename, errno); . . . }






struct stat sb;if (fstat(fd, &sb) < 0) { os_log_fault(general_log, "Failed to fstat %{public}s - %{errno}d", filename, errno); . . . }






struct stat sb;if (fstat(fd, &sb) < 0) { os_log_fault(general_log, "Failed to fstat %{public}s - %{errno}d", filename, errno); . . . }

os_log_info(time_log, "status for file %{public}s, atime:%{timespec}.*P, mtime:%{timespec}.*P, ctime:%{timespec}.*P",

filename, sizeof(struct timespec), &sb.st_atimespec, sizeof(struct timespec), &sb.st_mtimespec, sizeof(struct timespec), &sb.st_ctimespec);

Activity API Improvements


Activities are now objects that can be stored and re-used


Activities are now objects that can be stored and re-used• Direct control of activity relationships during creation


Activities are now objects that can be stored and re-used• Direct control of activity relationships during creation

New API to auto-scope activities within your code

Improved Activity APIs

os_activity_create Creates a new activity object

os_activity_scope Makes an execution block a part of an activity

os_activity_apply Invokes a block scoped to a given activity

os_activity_label_useraction Label an activity as a user action (UI-based activities)

NEW

/* * New Activity API Example */


os_activity_t init_activity = os_activity_create("Init", OS_ACTIVITY_CURRENT, OS_ACTIVITY_FLAG_DEFAULT);



os_activity_t verify_activity = os_activity_create("Verify", init_activity, OS_ACTIVITY_FLAG_DEFAULT);




if (isReady) { os_activity_scope(verify_activity); // All of the following work is done under “verification activity scope” . . .}




if (isReady) { os_activity_scope(verify_activity); // All of the following work is done under “verification activity scope” . . .} // This is not part of that activity




if (isReady) { os_activity_scope(verify_activity); // All of the following work is done under “verification activity scope” . . .} // This is not part of that activity

os_activity_apply(init_activity, ^{ // do some work based on the “init_activity” . . .});

Console NEW

Console

View live content from a system

NEW

Console

View live content from a systemOpen log archives

NEW

Console

View live content from a systemOpen log archivesNew Activity centric view of logging and tracing

NEW

Console

View live content from a systemOpen log archivesNew Activity centric view of logging and tracingAdvanced filtering and searching

NEW

Console

View live content from a systemOpen log archivesNew Activity centric view of logging and tracingAdvanced filtering and searchingDevice support

NEW

log Command Line Tool NEW

log Command Line Tool

Same functionality as Console from the command line

NEW


Same functionality as Console from the command lineStream live log messages

NEW



$ log stream

NEW



$ log stream$ log stream --predicate 'eventMessage contains "my message"'

NEW



$ log stream$ log stream --predicate 'eventMessage contains "my message"'Display a log file or archive

NEW




$ log show system_logs.logarchive

NEW




$ log show system_logs.logarchiveEnable debug for your subsystem on macOS

NEW




$ log show system_logs.logarchiveEnable debug for your subsystem on macOS

$ log config --mode "level:debug" --subsystem com.mycorp.myapp

NEW

Coming Soon NEW

Coming Soon

Tools for accessing new log information from 10.11 are coming soon

NEW

Coming Soon

Tools for accessing new log information from 10.11 are coming soonBut… in the meantime

NEW

Coming Soon

Tools for accessing new log information from 10.11 are coming soonBut… in the meantime

$ xcrun simctl spawn booted log show system_logs.logarchive

NEW

Best Practices

Logging Etiquette

Logging Etiquette

Ensure messages contain only information useful for debugging

Logging Etiquette

Ensure messages contain only information useful for debuggingLet us do the formatting for you—leverage built-in formatters

Logging Etiquette

Ensure messages contain only information useful for debuggingLet us do the formatting for you—leverage built-in formattersAvoid creating wrapper functions for os_log* APIs

Logging Etiquette

Ensure messages contain only information useful for debuggingLet us do the formatting for you—leverage built-in formattersAvoid creating wrapper functions for os_log* APIsLog only what you need from collections (Dictionaries, Arrays, etc.)

Logging Etiquette

Ensure messages contain only information useful for debuggingLet us do the formatting for you—leverage built-in formattersAvoid creating wrapper functions for os_log* APIsLog only what you need from collections (Dictionaries, Arrays, etc.)Avoid logging in tight code loops

Using os_log Family of APIs


Use os_log to log critical details to help debug issues


Use os_log to log critical details to help debug issuesUse os_log_info for additional info that will be captured during error or fault


Use os_log to log critical details to help debug issuesUse os_log_info for additional info that will be captured during error or faultUse os_log_debug for high-volume debugging during development


Use os_log to log critical details to help debug issuesUse os_log_info for additional info that will be captured during error or faultUse os_log_debug for high-volume debugging during developmentUse os_log_error to cause additional information capture from app


Use os_log to log critical details to help debug issuesUse os_log_info for additional info that will be captured during error or faultUse os_log_debug for high-volume debugging during developmentUse os_log_error to cause additional information capture from appUse os_log_fault to cause additional information capture from system

Gathering Logs

Using sysdiagnose NEW

Using sysdiagnose

sysdiagnose is the preferred method to capture data for bug reports

NEW

Using sysdiagnose

sysdiagnose is the preferred method to capture data for bug reports• Unified Logging data in system_logs.archive

NEW

Using sysdiagnose


You can use key-chord to trigger

NEW

Using sysdiagnose


You can use key-chord to triggersysdiagnose on Apple Watch will trigger on both Apple Watch and iPhone

NEW

Using sysdiagnose


You can use key-chord to triggersysdiagnose on Apple Watch will trigger on both Apple Watch and iPhoneTransfer from device using iTunes

NEW

Using sysdiagnose


You can use key-chord to triggersysdiagnose on Apple Watch will trigger on both Apple Watch and iPhoneTransfer from device using iTunesThis is the file to send to Apple (either Radar or Developer Technical Support)

NEW

Key-chords for sysdiagnose NEW

Mac OS Shift + Control + Option + Command + Period (.)

iOS Volume Up + Volume Down + Power Slight vibration on iPhone to indicate start

watchOS Press and hold Digital Crown + Side Button for 1 second A screen shot is triggered if not held long enough. Slight haptic to indicate start.

tvOS Play/Pause + Volume Down On older remote controls for Apple TV, must be held for 5 seconds

Deprecations

Deprecation of Legacy logging APIs


All ASL logging APIs are superseded by the new APIs


All ASL logging APIs are superseded by the new APIsNew APIs for searching new log data will not be made public this release


All ASL logging APIs are superseded by the new APIsNew APIs for searching new log data will not be made public this release• No equivalent asl_search functionality

os_activity_start Use os_activity_create and os_activity_scope / os_activity_apply

os_activity_end Use os_activity_create and os_activity_scope / os_activity_apply

os_activity_set_breadcrumb Use os_activity_label_useraction

os_trace_with_payload Use os_log*

Deprecated Activity APIs

Summary

Summary

The new Unified Logging system is

Summary

The new Unified Logging system is• Faster

Summary

The new Unified Logging system is• Faster• Easier to use

Summary

The new Unified Logging system is• Faster• Easier to use• Gives you more control

Summary

The new Unified Logging system is• Faster• Easier to use• Gives you more control

But requires using new APIs and new tools

Related Sessions

Fix Bugs Faster Using Activity Tracing WWDC 2014

More Information

https://developer.apple.com/wwdc16/721

Unified Logging and Activity Tracing

Documents

Transcript of Unified Logging and Activity Tracing