Data Strategies in Testing
-
Upload
paul-merrill -
Category
Software
-
view
320 -
download
0
Transcript of Data Strategies in Testing
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Why do we care about the data used by the system under test?
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Data StrategiesPAUL MERRILL
@dpaulmerrill
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
AgendaDefine Terms Identify Constraints Walk Through Examples
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Define Terms
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Define TermsData Strategy - the combination of code, procedure and infrastructure that affect how tests interact with data to stimulate the System Under Test (SUT).
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Define TermsSigned Data - data which has been modified so that it is easily recognizable.
For example, an account with a user name like “TestUser” or an email address like
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Define TermsConstraint - a characteristic that applies a force in a way that is limiting to an automated test or its environment.
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Define TermsConstraints factor down to:
Policy Cost Test Expectations Infrastructure People
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Identify Constraints
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Examples of ConstraintsShared Environment
– Cost – Policy – Infrastructure
Downtime – Policy – Cost – People
Technical Skills – Cost – People
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Examples of ConstraintsData Source Growth
– Cost – Policy – Infrastructure
Repeatable Tests – Policy – Test Expectations – People – Infrastructure
Data Variance – Policy
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Example Data Strategies
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Strategies Have 2 Parts:Creational Strategy - how and when data is created. Cleanup Strategy - the method by which a data source is returned to a benign state.
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Elementary ApproachCreation - Tests assume specific data in SUT - no creational strategy Clean up - No clean-up strategy, no management of data
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Elementary Approach
SUT
Execution 1
Execution 2
Execution 3
Data Source
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Elementary Approach
Pros:Simple No implementation (quick)
Cons:Tests are not repeatable No isolation of data Does not work on different systems
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Refresh Data Source ApproachCreation - Tests assume specific data in SUT Cleanup - Data managed by refreshing the data source prior to test execution
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Refresh Data Source Approach
SUT
Execution 1
Data Source
Refresh
Execution 2
Refresh
Execution 3
Refresh
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Refresh Data Source ApproachPros:
Clean test bed (data-wise) per test Uniform, one-size-fits-all process No variance of data Less time/cost to implement Total control of data
Cons:Timely Requires Downtime Blocks shared environments No variance of data Stale data over time Doesn’t work on different systems
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Selfish Data GenerationCreation - Tests create the data they need in SUT Clean up - No management of data after generation May be random (e.g. via Faker) Create with batch process or in test Create via SUT or directly in Data Source Create In-test or pre-test
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Selfish Data Generation
SUT
Execution 1
Execution 2
Execution 3
Data Source
Generate Data
Generate Data
Generate Data
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Selfish Data GenerationPros:
Unique data per test Does not block shared environments No downtime No stale data Tests can run anywhere Tests become repeatable
Cons:Unchecked growth of data source Requires understanding of data model Tests need to know about data
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Data Generation & Batch CleanupCreation - Data generated for each test Cleanup - Data cleaned up after test run Clean up may happen in data source or via SUT
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Data Generation & Batch Cleanup
SUT
Execution 1
Execution 2
Execution 3Data
Source
Generate Data
Generate Data
Generate Data
Batch Clean up
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Data Generation & Batch CleanupPros:
Unique data per test Does not block shared environments Minimal downtime No stale data Growth of data source kept in check Tests run anywhere Tests are repeatable
Cons:Requires deep understanding of data model Must know what to clean Significant scaffolding must be created
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Data Generation & Immediate Cleanup
Creation - tests generate their own data Clean up - for each test case immediately after execution Uses a TearDown method supplied by the framework
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Data Generation & Immediate Cleanup
SUT
Execution 1
Execution 2
Execution 3
Data Source
Generate Data
Generate Data
Generate Data
Clean up
Clean up
Clean up
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Data Generation & Immediate CleanupPros:
Tests are repeatable Unique data per test Does not block shared environments Minimal downtime No stale data Growth of data source kept in check No data left over from other tests Tests run anywhere
Cons:Requires deep understanding of data model Must know what to clean Significant scaffolding must be created
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Creational StrategiesElementary Refreshed Data
SourceData Generation
Shared EnvironmentLow Downtime
Repeatable TestsData Variance
Minimal Data GrowthSignificant Technical Skills
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Cleanup StrategiesElementary (none)
Refresh Data Source
Batch Immediate (Signed, Marked or Persisted)
Shared EnvironmentLow DowntimeRepeatable TestsStrong Technical Skills
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
How do we know what data to clean up?
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Signed DataRecord Name: “test_21323” Field Name: “Record Name” Format: “test_<random5digs>” Persist in: – a data source – code
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Marked DataAn “isTestData” field or Data that is otherwise known to be test data because of its format or type, for example: – A “test user” created it – Linked to known test denotation
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Internally Persisted DataKeep a list in memory: – userIds:
• 12312 • 4322 • 23423
– emailIds: • 12312 • 423423 • 234
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Externally Persisted DataKeep a record in a data source: – userIds:
• 12312 • 4322 • 23423
– emailIds: • 12312 • 423423 • 234
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
In ReviewDefined Terms Identified Constraints Walked Through Examples
www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved
Thank you!PAUL MERRILL
@dpaulmerrill