Massive Stochastic Testing of SQLMassive Stochastic Testing of SQL
Don SlutzDon SlutzMicrosoft ResearchMicrosoft Research
Presented ByPresented By
Manan ShahManan Shah
Testing a DatabaseTesting a Database
Two Broad Aspects:Two Broad Aspects: Testing the correctness of the SQL EngineTesting the correctness of the SQL Engine Testing the correctness of the SQL outputTesting the correctness of the SQL output
Focus of this paper:Focus of this paper: Testing the correctness of the SQL Engine.Testing the correctness of the SQL Engine.
Test Coverage ProblemTest Coverage Problem
All possible SQL
Used by customers
SQL test library
Detectable software bugs
BackgroundBackground SQL test groups focus on deterministic testing to cover
individual features of the language.
Typical SQL test libraries require an estimated ½ person-hour per statement to compose.
Software engineering requires a substantial investment
Commercial SQL vendors with tight schedules tend to use a more ad hoc process.
MotivationMotivation Deterministic testing of SQL database systems is human
intensive.
The input domain, all SQL statements, from any number of users, with all states of the database, is gigantic.
These test libraries cover an important, but tiny, fraction of the SQL input domain.
Large increases in test coverage must come from automating the generation of tests.
Our AimOur Aim
All possible SQL
Used by customers
SQL test library
Detectable software bugs
IntroductionIntroduction This paper describes a method to rapidly create a very large
number of SQL statements without human intervention.
The SQL statements are generated stochastically.
Stochastic testing has the advantage that the quality of the tests improves as the test size increases
Introduces idea of cross-validation of query results against existing SQL implementations
RAGS (Random Generation Of SQL) is currently used by the Microsoft SQL Server testing group.
SQL GrammarSQL Grammar
SELECT FROMSELECT FROM
WHERE WHERE
selectpart frompart
[TOP term] [DISTINCT | ALL] selectExpression [,...]
* | expression [[AS] columnAlias] | tableAlias.*
tableExpression [,...]
{[schemaName.] tableName] [{{LEFT | RIGHT} [OUTER] | [INNER] | CROSS | NATURAL} JOIN tableExpression
expression
andcondition [OR andcondition]
operand [conditionRightHandSide]| NOT condition| EXISTS (select)
ExampleExample
Emp_noEmp_no NameName SexSex Dept.Dept. JobJob SalarySalary CommComm
11 JohnJohn MM R&DR&D Engg.Engg. 2500025000 25002500
22 MaryMary FF SalesSales ClerkClerk 1500015000 15001500
33 AndyAndy MM R&DR&D Mgr.Mgr. 5000050000 50005000
44 SamSam MM SalesSales Mgr.Mgr. 5000050000 50005000
55 PetePete MM R&DR&D ClerkClerk 1500015000 15001500
Stochastic generationStochastic generationSelect
name
salary
+
commfrom Table Emp
where
and
10000salary
>
“sales”Dept
=
SQL Statement GenerationSQL Statement Generation
RAGS follows the semantic rules of SQL.
Carries state information and directives on its walk down the tree and the results of stochastic outcomes as it walks up.
RAGS makes all its stochastic decisions at the last possible moment
The RAGS SystemThe RAGS System The RAGS approach is:
1. Greatly enlarge the shaded circle in the figure by stochastic SQL statement generation.
2. Make all aspects of the generated SQL statements configurable.
3. Experiment with configurations to maximize the bug detection rate.
The RAGS System The RAGS System cont…cont…
RAGS is an experiment to see how effective a million fold increase in the size of a SQL test library can be.
RAGS can be used to drive an SQL system and look for observable errors
The output of successful Select statements can be saved for regression testing.
The RAGS SystemThe RAGS System
Print Report
Config. file
DBMS1=SYS A
DBMS2=SYS B
100000 Stmts
65% Select
35%Update
Connect to DBMS’s
Read Table Schema
Loop
Generate SQL Stmt
Execute on SYS A
Execute on SYS B
Compare Results
Record Errors
SQL DBMS SYS A
Read configuration
SQL DBMS SYS B
Report:
100000 Stmts exec.
Stmt 156: error in SYS A
Stmt 765: error in SYS B
Components of RAGS SystemComponents of RAGS System The configuration file has several parameters: The configuration file has several parameters:
1. Frequency of occurrence of different statements1. Frequency of occurrence of different statements2. Limits (max no. of tables in a join, entries in group by, etc.)2. Limits (max no. of tables in a join, entries in group by, etc.)3. Frequency of occurrence of features (group by, order by)3. Frequency of occurrence of features (group by, order by)4. Execution Parameters (no. of rows to fetch per query)4. Execution Parameters (no. of rows to fetch per query)
When the RAGS program is started, it first reads the configuration file.
It then uses ODBC to connect to the first DBMS and read the schema information.
Execution of RAGSExecution of RAGS RAGS loops to generate SQL statements and optionally
execute them.
If the statement is executed on more than one system, the execution results are compared
At the end of the run, RAGS produces a report.
A utility is provided that compares the reports from several runs and summarizes the differences.
The comparison can be between different vendors or different versions of the same system (regression testing).
Validation IssuesValidation Issues If a SQL Select executes without errors, there is no easy
method to validate the returned values
The authors’ approach is to execute the same query on multiple vendor’s DBMSs and then compare the results.
First, the number of rows returned is compared
To avoid sorts, a special checksum over all the column values in all the rows is compared.
The method only works for SQL statements that will execute on more than one vendor’s database.
Example RAGS QueryExample RAGS Query
Execution FactsExecution Facts On a 200Mhz Pentium RAGS can generate 833 moderate size
SQL statements per second.
In one hour RAGS can generate 3 million different SQL statements
The starting random seed for a RAGS run can be specified in the configuration file.
If the starting seed is not specified, RAGS obtains a seed by hashing the time of day.
Testing ExperiencesTesting Experiences
Each of the 10 clients executed 2500 SQL statements in transactions that contained an average of 5 statements
86.1% of the statements executed without error, 13.8% had expected errors and 0.07% indicated possible bugs
Comparison TestsComparison Tests
Automatic Statement SimplificationAutomatic Statement Simplification
When a RAGS generated statement caused an error, the debugging process was difficult if the statement was complex
The offending statement can usually be vastly simplified by hand.
The simplification process itself was tedious, so RAGS was extended to simplify the statement automatically.
RAGS walks a parse tree for the statement and tries to remove terms in expressions and certain clauses (Where and Having).
VisualizationVisualization
To investigate the relationship between two metrics, a set of sample pairs is collected and analyzed.
RAGS presents an opportunity to scale up the size of such samples by several orders of magnitude.
It allows one to plot the sample points and visualize the relationship.
Execution time of V2 (sec)
Execution time of V1 (sec)0 0.1 0.2 0.3 0.4
0.4
0.3
0.2
0.1
SummarySummary RAGS is an experiment in massive stochastic testing of SQL
systems.
Its main contribution is to generate entire SQL statements stochastically
The problem of validating outputs remains a tough issue. Output comparisons for different vendor’s database systems proved to be extremely useful, but only for the small set of common SQL.
The outcome of the experiments was encouraging since RAGS could steadily generate errors in released SQL products.
Future WorksFuture Works SQL coverage can be extended to more data types, more DDL,
stored procedures, utilities, etc.
Robustness tests performed by stochastically generating a family of equivalent SQL statements and comparing their O/P
Testing with equivalent statements has the important advantage of a method to help validate the outputs.
The optimizer estimates of execution metrics, together with the measured execution metrics, can be compared.
Thank You!Thank You!
Top Related