At GitLab, we use different strategies to make assessments about the stability
or robustness of a feature by means of best practices such as staging
environments, feature flags, or canary testing. We also use testing
strategies such as A/B testing
to assess how users react to feature variants.
However, our short release cycles require testing and benchmarking approaches that
make it possible to prototype, test, and benchmark ideas quickly (ideally while
developing them). We need an approach that works on large code
bases, can help assess a feature before deployment to staging or
production, and provides data to support data-driven decision making.
To address this need, we developed the SourceWarp tool: a record-and-replay framework
for source code management systems. In this blog post, we will explain our motivation
for creating SourceWarp and explain how we use it to inform data-driven decision making within the GitLab platform.
Motivation: Data-driven decision making in the DevSecOps context
DevSecOps streamlines software development by allowing teams to ship features quickly
and providing short feedback cycles for customers. These short feedback cycles can be used to monitor the impact of
a feature from the time it is shipped and inform developers and product
managers about the success or failure of a given deployment.
GitLab, as a heterogeneous DevSecOps platform, acts as an integration point for
different CI/CD tools that often contribute
to user-facing functionality. For example, the vulnerability report,
which displays all detected vulnerabilities, is visible
as a single functionality, but the data in the report may come from a
number of different tools in various pipelines. The DevSecOps
platform collects and stores results in the backend database and keeps track of user actions on the
findings (through the UI or the API). A large portion of the automation in the platform
is built around or initiated by code changes where the
source code management system or Git respoitory basically holds the input data. In
order to test and benchmark new features for these systems effectively, the
testing and benchmarking approach needs to have some source code awareness.
We can use SourceWarp to achieve this. Let's dive in to a real-world example
of how we used SourceWarp to help make an informed decision about a product integration.
Case study: Advanced vulnerability tracking
As a DevSecOps platform, GitLab provides automation
centered around code changes, where the source code is stored in a source code
management system. SourceWarp uses a Git repository as input, which we use to
source test-input data to test and benchmark our newly developed feature.
In a record phase, SourceWarp extracts commits from the source history that are
relevant with respect to a given test criterion and generates a patch replay
sequence. In the monitor phase, SourceWarp replays the generated sequence on a
target system. These phases are executed while continuously monitoring the
DevSecOps platform to collect metrics and to generate a report that provides
the testing and benchmarking results.
We used SourceWarp to test and benchmark advanced vulnerability tracking,
which identifies and deduplicates vulnerabilities in a changing code base. In our
benchmarking and testing experiment, we let SourceWarp automatically sample patch
sequences from a slice of GitLab's source code repository history (2020-10-31
and 2020-12-31) and replay them on two target systems: One system had advanced
vulnerability tracking enabled, and the other one was using our old
vulnerability tracking approach.
After the application of every patch from the
patch sequence, SourceWarp collected metrics from the target system that
recorded the observed vulnerabilities. We observed that our vulnerability
tracking approach was 30% more effective than traditional
vulnerability tracking where <file, line number> are used to identify the
location of a vulnerabilty. This means that advanced vulnerabiilty tracking
reduces the manual effort of auditing vulnerabilities by 30%.
In addition, we
observed that with an increasing number of source code changes, the deduplication
effectiveness of vulnerability tracking increases. Looking at the relatively
short timeframe from 2020-10-31 to 2020-12-31, the deduplication effectivness
increased from 11% to 30%, which suggests that the effectiveness increases over
time as the source code evolves.
SourceWarp performed this experiment in an automated and reproducible way, and
provided data that was helpful in making an informed decision about the product
integration of vulnerability tracking.
Where to find more SourceWarp information
The SourceWarp approach is detailed in our research paper, "SourceWarp: A scalable, SCM-driven testing and benchmarking approach to support data-driven and agile decision making for CI/CD tools and DevSecOps platforms," which will be presented at the 4th ACM/IEEE International Conference on Automation of Software Test (AST 2023).
The SourceWarp testing and benchmarking tool is implemented in Ruby and is open source (MIT license).
The README.md provides information about the tool setup and implementation.
You can also see it in action in the demo below.
Useful Links
Cover image by Jason Corey on Unsplash
