Introduction to Benchmark

41
Introduction to Benchmark Database Systems DataLab, CS, NTHU Spring, 2022

Transcript of Introduction to Benchmark

Page 1: Introduction to Benchmark

Introduction to Benchmark

Database SystemsDataLab, CS, NTHU

Spring, 2022

Page 2: Introduction to Benchmark

• VanillaBench Project

• Introduction to VanillaBench

• Setting Benchmark Configurations

• Starting Up Server for Benchmarking

• Running Benchmark Client

• Assignment 2

Outline

Page 3: Introduction to Benchmark

• VanillaBench Project

• Introduction to VanillaBench

• Setting Benchmark Configurations

• Starting Up Server for Benchmarking

• Running Benchmark Client

• Assignment 2

Outline

Page 4: Introduction to Benchmark

VanillaBench

• VanillaBench is a project designed for automatically benchmarking VanillaCore

• It contains several benchmark procedures• It also has a lot of adjustable testing parameters

[Icon Source]

Page 5: Introduction to Benchmark

Two Main Methods

BENCHMARKSERVER-StartUp

BENCHMARKCLIENT-App

Page 6: Introduction to Benchmark

JDBC / SP ?

JDBC SP

Query Parameter

Page 7: Introduction to Benchmark

Create SP-- Insert userCREATE PROCEDURE insertuser(uname VARCHAR(50), ukarma INT) LANGUAGE SQLAS $$

INSERT INTO users(name, karma) VALUES (uname, ukarma);$$;

-- Insert postCREATE PROCEDURE insertpost(uname VARCHAR(50), post TEXT) LANGUAGE SQLAS $$

INSERT INTO posts(text, "authorId")VALUES (post, (SELECT id FROM users WHERE name = uname));

$$;

Page 8: Introduction to Benchmark

• VanillaBench Project

• Introduction to VanillaBench

• Setting Benchmark Configurations

• Starting Up Server for Benchmarking

• Running Benchmark Client

• Assignment 2

Outline

Page 9: Introduction to Benchmark

Clone the Project First

• The code of VanillaBench has been pushed to vanilladb repository

• All you need is to clone from the remote repository

• You can clone from here:

• https://shwu10.cs.nthu.edu.tw/courses/databases/2022-spring/db22-assignment-2

> git clone

Page 10: Introduction to Benchmark

1.

2.

3.

Import Project(1/3)

Page 11: Introduction to Benchmark

Import Project(2/3)

4.

5.

6.

Page 12: Introduction to Benchmark

Import Project(3/3)

7.

8.

9.

10.

Page 13: Introduction to Benchmark

Finish Import Projects

Page 14: Introduction to Benchmark

Benchmark Setting

• Benchmark project also has its own set of properties files

Page 15: Introduction to Benchmark
Page 16: Introduction to Benchmark

Use JDBC or stored procedures

Page 17: Introduction to Benchmark

Benchmark report path

Page 18: Introduction to Benchmark

• VanillaBench Project

• Introduction to VanillaBench

• Setting Benchmark Configurations

• Starting Up Server for Benchmarking

• Running Benchmark Client

• Assignment 2

Outline

Page 19: Introduction to Benchmark

Two Main Methods

BENCHMARKSERVER-StartUp

BENCHMARKCLIENT-App

Page 20: Introduction to Benchmark

Starting Up Server (1/2)• To benchmark a

VanillaDB server, you need to start up the server in another entry point

1.

2.3.

4.

5.

Page 21: Introduction to Benchmark

Starting Up Server (2/2)

• Add the Database Directory Name

• You also need to add one more VM argument for benchmarking.

You can copy those arguments from here.

6.

7.

Page 22: Introduction to Benchmark

Server Messages

You should see similar messages if nothing is wrong.

Page 23: Introduction to Benchmark

• VanillaBench Project

• Introduction to VanillaBench

• Setting Benchmark Configurations

• Starting Up Server for Benchmarking

• Running Benchmark Client

• Assignment 2

Outline

Page 24: Introduction to Benchmark

Two Main Methods

BENCHMARKSERVER-StartUp

BENCHMARKCLIENT-App

Page 25: Introduction to Benchmark

The Workflow of A Client

App

loadTestbed() benchmark()

Create

Load Testbed Start Benchmarking

Depends on the Argumentorg.vanilladb.bench.VanillaBench

Page 26: Introduction to Benchmark

JDBC / SP ?

JDBC SP

Query Parameter

CheckDatabaseJdbcJob

TestbedLoaderJdbcJob

ReadItemTxnJdbcJob

TestbedLoaderProc

CheckDatabaseProc

ReadItemTxnProc

Page 27: Introduction to Benchmark

The Workflow of A Client

App

loadTestbed() benchmark()

Create

Load Testbed Start Benchmarking

Depends on the Argumentorg.vanilladb.bench.VanillaBench

Page 28: Introduction to Benchmark

Loading Testbed

loadTestbed()

Connect to server and execute:

TestbedLoader

Page 29: Introduction to Benchmark

Running Client• To run clients, create a run configuration for it

Page 30: Introduction to Benchmark

Arguments• We also need to set some arguments

• Program Arguments

• [Action]

• 1 : Load Test-Bed

• 2 : Launch Benchmark

• VM Arguments-Djava.util.logging.config.file=target/classes/java/util/logging/logging.properties-Dorg.vanilladb.bench.config.file=target/classes/org/vanilladb/bench/vanillabench.properties -Dorg.vanilladb.core.config.file=target/classes/org/vanilladb/core/vanilladb.properties

Page 31: Introduction to Benchmark

Loading Testbed

Page 32: Introduction to Benchmark

Client Messages

Page 33: Introduction to Benchmark

The Workflow of A Client

App

loadTestbed() benchmark()

Create

Load Testbed Start Benchmarking

Depends on the Argumentorg.vanilladb.bench.VanillaBench

Page 34: Introduction to Benchmark

Starting Benchmark

benchmark()

Create

RTEs

Remote Terminal Emulator

Emulates a remote terminal, executing a sequence of transactions

Page 35: Introduction to Benchmark

Server & Client

BENCHMARKSERVER

RTE

RTE RTE

Page 36: Introduction to Benchmark

RTE’s Life Cycle

run()

stopBenchmark()end

executeTxCycle()

org.vanilladb.bench.rte.RemoteTerminalEmulator

Page 37: Introduction to Benchmark

Executing a TxexecuteTxCycle()

getTxExecutor(type)

executor

executor.execute() start to record execution time

executeTxn()

getJdbcExecutor().execute(…) callStoredProc(…)

Depends on JDBC / SP

RTE

TransactionExecutor

Page 38: Introduction to Benchmark

Benchmarking

Page 39: Introduction to Benchmark

Client Messages

Page 40: Introduction to Benchmark

Assignment 2

Page 41: Introduction to Benchmark

Q&A

• If you have any problem, you could check here first

• https://shwu10.cs.nthu.edu.tw/courses/databases/2022-spring/faq

• If your problem is very unique, feel free to send us an email