Introducing taocpp/jsonhttps://github.com/taocpp/json

Daniel Frey 2016-11-10

Disclaimer• Opinions expressed are solely my own and do

not express the views or opinions of my employer

• All mistakes are mine, please point them out

• When taking a note for a question, please write down the slide number!

2

Origin• Authors of the PEGTL and taocpp/json:

Dr. Colin Hirsch and Daniel Frey

• Originally a benchmark for the PEGTLhttps://github.com/ColinH/PEGTL

• Benchmarked against 40 other libraries https://github.com/miloyip/nativejson-benchmark

• Inspired by Niels Lohmann’s JSON library https://github.com/nlohmann/json

3

Our use-cases• REST APIs

• Logging: Structured data, supports ELK stack

• Several thousand JSON value constructions and serializations per second

• Main development driven by demand

• Used in production code

4

Type

— tao::json::type

JSO

N v

alue

json::value• Contains a type and a

C++ union to store data

• Query type: v.type()

• Basic JSON types

NULL_

ARRAY

OBJECT

BOOLEAN

STRING

(Number)

5

Type

— tao::json::type

JSO

N v

alue

Type• Contains a type and a

C++ union to store data

• Query type: v.type()

• Basic JSON types

• Numbers come in three flavours

NULL_N

umbe

r

ARRAY

OBJECT

SIGNED

UNSIGNED

DOUBLE

BOOLEAN

STRING

6

Type

— tao::json::type

JSO

N v

alue

Type• First extension:

A plain old raw pointer

• Rarely used…

• …but allows important optimisations!

• Non-owning, dumb. Like… really dumb!

NULL_N

umbe

r

ARRAY

OBJECT

SIGNED

UNSIGNED

DOUBLE

BOOLEAN

STRING

RAW_PTR

7

Type

— tao::json::type

JSO

N v

alue

DataNULL_

Num

ber

ARRAY

OBJECT

SIGNED

UNSIGNED

DOUBLE

BOOLEAN

STRING

RAW_PTR

std::uint64_t

double

std::string

std::vector<value>

std::map<std::string, value>

std::int64_t

bool

const value*

-

8

Type

— tao::json::type

JSO

N v

alue

GetterNULL_

Num

ber

ARRAY

OBJECT

SIGNED

UNSIGNED

DOUBLE

BOOLEAN

STRING

RAW_PTR

v.get_unsigned()

v.get_double()

v.get_string()

v.get_array()

v.get_object()

v.get_signed()

v.get_boolean()

v.get_raw_ptr()

9

Type

— tao::json::type

JSO

N v

alue

Status• Second extension:

Other statuses, part 1

• Default-constructed values do not have a type assigned. The type/status is UNINITIALIZED

NULL_N

umbe

r

ARRAY

OBJECT

SIGNED

UNSIGNED

DOUBLE

BOOLEAN

STRING

UNINITIALIZED

RAW_PTR

10

Type

— tao::json::type

JSO

N v

alue

Status• Second extension:

Other statuses, part 2

• When a value is destructed, moved, … the type/status is set to DISCARDED (sometimes)

NULL_N

umbe

r

ARRAY

OBJECT

SIGNED

UNSIGNED

DOUBLE

BOOLEAN

STRING

UNINITIALIZED

RAW_PTR

11

DISCARDED

LoggingLOG(INFO, 1234, “Hello, world!”);

// LOG(LVL,ID,…) macro expands to: if(tao::log::is_active(LVL, ID)) { tao::json::value v = __VA_ARGS__; tao::log::write(LVL, ID, v); }

12

Value constructionjson::value v = “Hello, world!”;

13

Value constructionjson::value v = “Hello, world!”; json::value v = true; json::value v = 42u; json::value v = -42; json::value v = 1.23;

14

Value constructionjson::value v = “Hello, world!”; json::value v = true; json::value v = 42u; json::value v = -42; json::value v = 1.23; json::value v = json::null; json::value v = json::empty_array; json::value v = json::empty_object;

15

Array constructionjson::value v = json::empty_array; v.emplace_back(true); v.emplace_back(42); v.emplace_back(“Hello, world!”);

16

Array constructionjson::value v = json::array({ true, 42, “Hello, world!” });

• Think std::initializer_list<value>.

• Reality is somewhat more complicated.

17

Object constructionjson::value v = json::empty_object; v.emplace(“foo”, true); v.emplace(“bar”, 42); v.emplace(“baz”, “Hello, world!”);

v[“foo”][“bar”]; // will throw v[“foo”] = 123u; v[“x”][“y”] = “nice”;

18

Object constructionjson::value v = { { “foo”, true }, { “bar”, 42 }, { “baz”, “Hello, world!” } };

• Keys are always strings, values are anything a direct construction of value accepts.

• No duplicate keys are allowed.

19

Object constructionjson::value v = { { “foo”, true }, { “bar”, 42 }, { “baz”, “Hello, world!” } };

• Keys are always strings, values are anything a direct construction of value accepts.

• No duplicate keys are allowed.

20

Object constructionjson::value v = { { “Some dwarfs”, { { “Thorin”, “Leader” }, { “Kili”, “Nephew” }, { “Balin”, “Third-cousin” }, { “Dori”, “Remote kinsman” }, { “Bifur”, “From Moria” } } } };

21

Object constructionjson::value dwarfs = { { “leader”, “Thorin” }, { “nephews”, json::array({ “Kili”, “Fili” })}, { “third_cousins”, json::array({ “Balin”, “Dwalin”, “Oin”, “Gloin” })}, … };

22

LoggingLOG(INFO, 1234, “Hello, world!”);

23

LoggingLOG(INFO, 1234, { { “foo”, true }, { “bar”, 42 }, { “baz”, “Hello, world!” } });

24

Loggingstruct user { bool is_human; std::string name; unsigned age; };

user u = …;

25

LoggingLOG(INFO, 1234, { { “msg”, “add user” }, { “user”, { { “is_human”, u.is_human }, { “name”, u.name }, { “age”, u.age } } } });

26

Loggingtemplate<> struct traits<user> { static void assign( value& v, const user& u) { v = { { “is_human”, u.is_human }, { “name”, u.name }, { “age”, u.age } }; } };

27

LoggingLOG(INFO, 1234, { { “msg”, “add user” }, { “user”, u } });

28

Loggingtemplate<> struct traits<user> { static void assign( value& v, const user& u ); static const char* default_key; };

29

LoggingLOG(INFO, 1234, { { “msg”, “add user” }, u });

30

LoggingLOG(INFO, 1234, { { “msg”, “add user” }, u });

LOG(INFO, 1235, { { “msg”, “assign manager” }, u, { “manager”, u2 } });

31

Traitstemplate<> struct traits<user> { static user as(const value& v); };

auto u = v.as<user>();

32

Traitstemplate<> struct traits<user> { static user as(const value& v) { return user( v.at("is_human").get_boolean(), v.at("name").get_string(), v.at("age").as<unsigned>() ); } }; // other options under consideration

33

Traits

• Construct a value from any T

• Convert a value to any T

• Modify default behaviour

• value is actually basic_value<traits>

34

SAX

… …

SAX producers

sax::from_value

sax::from_string

sax::parse_file

SAX API

null() boolean(v) number(v) string(v)

begin_array() element() end_array()

begin_object() key(v) member()

end_object()

SAX consumers

sax::to_value

sax::to_stream

sax::to_pretty_stream

SAX schema validation

nlohmann::to_value nlohmann::from_value

sax::tee

35

SAX producertemplate<typename Consumer> void from_value(const value& v, Consumer& c) { switch(v.type()) { case type::UNINITIALIZED: throw …; case type::DISCARDED: throw …; case type::NULL_: c.null(); return; case type::BOOLEAN: c.boolean(v.get_boolean()); return; case type::SIGNED: c.number(v.get_signed()); return; case type::UNSIGNED: c.number(v.get_unsigned()); return; case type::DOUBLE: c.number(v.get_double()); return; case type::STRING: c.string(v.get_string()); return; … } }

36

SAX producertemplate<typename Consumer> void from_value(const value& v, Consumer& c) { switch(v.type()) { … case type::ARRAY: c.begin_array(); for(const auto& e : v.get_array()) { from_value(e, c); c.element(); } c.end_array(); return; … } }

37

SAX producertemplate<typename Consumer> void from_value(const value& v, Consumer& c) { switch(v.type()) { … case type::OBJECT: c.begin_object(); for(const auto& e : v.get_object()) { c.key(e.first); from_value(e.second, c); c.member(); } c.end_object(); return; … } }

38

SAX producertemplate<typename Consumer> void from_value(const value& v, Consumer& c) { switch(v.type()) { … case type::RAW_PTR: if(const auto* p = v.get_raw_ptr()) { from_value(*p, c); } else { c.null(); } return; } throw …; }

39

SAX

… …

SAX producers

sax::from_value

sax::from_string

sax::parse_file

SAX API

null() boolean(v) number(v) string(v)

begin_array() element() end_array()

begin_object() key(v) member()

end_object()

SAX consumers

sax::to_value

sax::to_stream

sax::to_pretty_stream

SAX schema validation

nlohmann::to_value nlohmann::from_value

sax::tee

40

SAX consumerclass to_stream { private: std::ostream& os; bool first = true; void next() { if(!first) os << ’,’; } public: explicit to_stream(std::ostream& os) noexcept : os(os) {} … };

41

SAX consumerclass to_stream { … void null() { next(); os << “null”; } void boolean(const bool v) { next(); os << (v ? “true” : “false”); } void number(const std::int64_t v) { next(); os << v; } void number(const std::uint64_t v) { next(); os << v; } void number(const double v) { next(); os << v; } void string(const std::string& v) { next(); os << ’”’ << escape(v) << ’”’; } … };

42

SAX consumerclass to_stream { … void begin_array() { next(); os << ’[’; first = true; } void element() { first = false; } void end_array() { os << ’]’; } void begin_object() { next(); os << ’{’; first = true; } void key(const std::string& v) { string(v); os << ’:’; first = true; } void member() { first = false; } void end_object() { os << ’}’; } };

43

SAX

… …

SAX producers

sax::from_value

sax::from_string

sax::parse_file

SAX API

null() boolean(v) number(v) string(v)

begin_array() element() end_array()

begin_object() key(v) member()

end_object()

SAX consumers

sax::to_value

sax::to_stream

sax::to_pretty_stream

SAX schema validation

nlohmann::to_value nlohmann::from_value

sax::tee

44

SAX decoupling• DOM-less usage, e.g., parse and pretty print a

file via SAX for extremely large JSON files

• Parse and generate binary formats (BSON, BJSON, UBJSON, …)

• Combine with other JSON libraries

• With tee, feed events to multiple consumers

45

46

0.Overall

20% 40% 60% 80% 100%

ArduinoJson(C++)

CAJUN(C++)

ccan/json(C)

cJSON(C)

Configuru(C++11)

dropbox/json11(C++11)

gason(C++11)

hjiang/JSON++(C++)

Jansson(C)

JeayeSON(C++14)

jsmn(C)

JSONSpirit(C++)

JSONVoorhees(C++)

json-c(C)

JsonBox(C++)

jsoncons(C++)

JsonCpp(C++)

JVar(C++)

Jzon(C++)

mikeando/FastJson(C++)

nbsdx_SimpleJSON(C++11)

Nlohmann(C++11)

Parson(C)

PicoJSON(C++)

RapidJSON(C++)

RapidJSON_AutoUTF(C++)

RapidJSON_FullPrec(C++)

RapidJSON_Insitu(C++)

sajson(C++)

Sheredomjson.h(C)

SimpleJSON(C++)

taocpp/json&Nlohmann(C++11)

taocpp/json(C++11)

tunnuz/JSON++(C++)

ujson(C++)

ujson4c(C)

Vinenthz/libjson(C)

YAJL(C)

69%69%

82%82%

89%89%

69%69%

99%99%

87%87%

55%55%

55%55%

87%87%

52%52%

58%58%

69%69%

88%88%

88%88%

65%65%

95%95%

90%90%

88%88%

74%74%

90%90%

34%34%

96%96%

83%83%

87%87%

93%93%

93%93%

100%100%

93%93%

86%86%

82%82%

67%67%

100%100%

100%100%

74%74%

94%94%

63%63%

88%88%

86%86%

Result

47

1.Parse

0 150 300 450 600

ArduinoJson(C++)

CAJUN(C++)

ccan/json(C)

cJSON(C)

Configuru(C++11)

dropbox/json11(C++11)

gason(C++11)

hjiang/JSON++(C++)

Jansson(C)

JeayeSON(C++14)

jsmn(C)

JSONSpirit(C++)

JSONVoorhees(C++)

json-c(C)

JsonBox(C++)

jsoncons(C++)

JsonCpp(C++)

JVar(C++)

Jzon(C++)

mikeando/FastJson(C++)

nbsdx_SimpleJSON(C++11)

Nlohmann(C++11)

Parson(C)

PicoJSON(C++)

RapidJSON(C++)

RapidJSON_AutoUTF(C++)

RapidJSON_FullPrec(C++)

RapidJSON_Insitu(C++)

sajson(C++)

Sheredomjson.h(C)

SimpleJSON(C++)

strdup(C)

taocpp/json&Nlohmann(C++11)

taocpp/json(C++11)

tunnuz/JSON++(C++)

ujson(C++)

ujson4c(C)

Vinenthz/libjson(C)

YAJL(C)

555555

588588

3030

2727

4949

4848

66

143143

6565

8181

343343

8383

253253

7272

269269

5353

7171

6868

159159

2020

155155

4141

4545

100100

77

1717

1313

77

77

1919

5050

00

5353

3030

174174

2929

88

5656

5757

Time(ms)

48

1.Parse

0 2,000,000 4,000,000 6,000,000 8,000,000

ArduinoJson(C++)

CAJUN(C++)

ccan/json(C)

cJSON(C)

Configuru(C++11)

dropbox/json11(C++11)

gason(C++11)

hjiang/JSON++(C++)

Jansson(C)

JeayeSON(C++14)

jsmn(C)

JSONSpirit(C++)

JSONVoorhees(C++)

json-c(C)

JsonBox(C++)

jsoncons(C++)

JsonCpp(C++)

JVar(C++)

Jzon(C++)

mikeando/FastJson(C++)

nbsdx_SimpleJSON(C++11)

Nlohmann(C++11)

Parson(C)

PicoJSON(C++)

RapidJSON(C++)

RapidJSON_AutoUTF(C++)

RapidJSON_FullPrec(C++)

RapidJSON_Insitu(C++)

sajson(C++)

Sheredomjson.h(C)

SimpleJSON(C++)

strdup(C)

taocpp/json&Nlohmann(C++11)

taocpp/json(C++11)

tunnuz/JSON++(C++)

ujson(C++)

ujson4c(C)

Vinenthz/libjson(C)

YAJL(C)

4545

6,248,3786,248,378

279,061279,061

263,589263,589

285,815285,815

579,229579,229

943943

877,851877,851

379,444379,444

974,012974,012

99

831,566831,566

1,477,9801,477,980

571,890571,890

1,427,2861,427,286

311,812311,812

425,949425,949

85,10485,104

1,719,4361,719,436

6868

2,208,2012,208,201

290,371290,371

594,122594,122

1,648,3761,648,376

115115

115115

115115

104104

1212

66

805,471805,471

66

825,699825,699

206,072206,072

1,729,5621,729,562

285,984285,984

3131

560,910560,910

768,299768,299

AllocCount

49

2.Stringify

0 70 140 210 280

ArduinoJson(C++)

CAJUN(C++)

ccan/json(C)

cJSON(C)

Configuru(C++11)

dropbox/json11(C++11)

gason(C++11)

hjiang/JSON++(C++)

Jansson(C)

JeayeSON(C++14)

JSONSpirit(C++)

JSONVoorhees(C++)

json-c(C)

JsonBox(C++)

jsoncons(C++)

JsonCpp(C++)

JVar(C++)

Jzon(C++)

mikeando/FastJson(C++)

nbsdx_SimpleJSON(C++11)

Nlohmann(C++11)

Parson(C)

PicoJSON(C++)

RapidJSON(C++)

RapidJSON_AutoUTF(C++)

RapidJSON_FullPrec(C++)

RapidJSON_Insitu(C++)

Sheredomjson.h(C)

SimpleJSON(C++)

strdup(C)

taocpp/json&Nlohmann(C++11)

taocpp/json(C++11)

tunnuz/JSON++(C++)

ujson(C++)

Vinenthz/libjson(C)

YAJL(C)

5050

161161

8585

5757

204204

8686

7272

279279

8787

7272

115115

8585

2121

207207

9595

135135

5757

2020

7777

149149

9696

164164

105105

88

1616

99

1010

6161

224224

00

2222

1919

172172

2121

8686

9797

Time(ms)

50

3.Prettify

0 60 120 180 240

ArduinoJson(C++)

ccan/json(C)

cJSON(C)

Configuru(C++11)

gason(C++11)

Jansson(C)

JSONSpirit(C++)

JSONVoorhees(C++)

json-c(C)

JsonBox(C++)

jsoncons(C++)

Jzon(C++)

nbsdx_SimpleJSON(C++11)

Nlohmann(C++11)

Parson(C)

RapidJSON(C++)

RapidJSON_AutoUTF(C++)

RapidJSON_FullPrec(C++)

RapidJSON_Insitu(C++)

Sheredomjson.h(C)

taocpp/json&Nlohmann(C++11)

taocpp/json(C++11)

ujson(C++)

Vinenthz/libjson(C)

YAJL(C)

127127

9898

5858

210210

9292

9595

149149

182182

3131

194194

9696

3333

147147

114114

166166

1010

3030

1010

1212

6767

3030

2727

2424

8686

113113

Time(ms)

Thank you!https://github.com/taocpp/json

Questions?