SemVer versioning: how we handled it with linear interval arithmetic

The semantic versioning (SemVer) specification can be
considered the de-facto standard for tracking software states during its
evolution. Unfortunately, in reality many languages/ecosystems practice "SemVer versioning" and have not adopted
the standard as-is; instead we can find many different semantic versioning
flavors that are not necessarily compatible with the original SemVer spec. SemVer Versioning has
led to the creation of a variety of different semantic versioning schemes.

GitLab provides a Dependency Scanning (DS)
feature that automatically detects vulnerabilities in the dependencies of a
software project for a variety of different languages. DS relies on the
GitLab Advisory Database
that is updated on a daily basis providing information about
vulnerable packages that is expressed in the package-specific (native)
semantic version dialect. GitLab also recently launched an Open Source Edition of the GitLab Advisory Database.

At GitLab we use a semi-automated process for advisory generation: we extract
advisory data that includes package names and vulnerable versions from
data-sources such as NVD and generate advisories that
adhere to the GitLab advisory format before they are curated and stored in our
GitLab Advisory Database.

The plethora of SemVer versioning in the wild posed a major
challenge for the level of automation we could apply in the advisory generation
process: the different semantic version dialects prevented us from building
generic mechanisms around version matching, version verification (i.e., the
process of verifying whether or not versions are available on the relevant package
registry), fixed version inference etc. Moreover, since advisory generation
requires us to extract and update advisory data on scale from data-sources with
hundreds of thousands vulnerability entries, translating and/or verifying
versions by hand is not a viable, scalable solution.

Having a generic method to digest and process a variety of different SemVer versioning dialects was an important building block for automating large parts of the advisory generation process. This led to the development of
semver_dialects, a
utility that helps processing semantic versions in a generic, language-agnostic manner which
has been recently open-sourced (MIT) and published on rubygems.org.

Understand the SemVer spec

The SemVer spec is the de-facto standard for tracking states of software projects during their evolution
by associating unique, comparable version numbers to distinct states, and by
encoding semantic properties into the semantic version strings so that a version
change implicitly conveys information about the nature of the change.

A semantic version consists of a prefix (version core) and a suffix that hold
pre-release and/or build information. A version core consists of three numeric
components that are delimited by .:

• major: backwards-incompatible changes
• minor: new backwards-compatible functionality
• patch: backwards-compatible bug fixes

Considering a software project using SemVer, with two releases 1.0.0 and
1.0.1, by just looking at the change applied to the semantic version strings,
it is clear that 1.0.1 is a newer (more recent) release of the software, whereas version
1.0.0 is an older release. In addition, the version number 1.0.1
represents an improved state of the software as compared to version 1.0.0 which contained a bug
that has been fixed in version 1.0.1. This fix is signalled by the higher number of the patch version component.

Semantic version processing is particularly useful in the context of Dependency Scanning (DS). DS is the process of automatically detecting (and potentially fixing)
vulnerabilities related to the dependencies of a software project: dependencies
of a software project are checked against a set of configuration files (so
called advisories) that contain information about vulnerable dependencies;
advisories usually include the versions of the vulnerable dependency.
Vulnerable versions are usually expressed in terms of version intervals: for example this out-of-bounds read vulnerability for the Python tensorflow package contains information about the vulnerable version by listing the four version intervals below:

1. up to 2.1.4
2. from 2.2.0 up to 2.2.3
3. from 2.3.0 up to 2.3.3
4. from 2.4.0 up to 2.4.2

While SemVer is very concise and clear about the syntax and semantic of
semantic versions, it does not specify how to express and represent semantic
version constraints. In addition, SemVer is purposefully simplistic to foster
its adoption. In practice it seems as if many ecosystems required features that
go beyond SemVer which led to the development of many SemVer versioning flavours as well
as a variety of different native constraint matching syntaxes, some of which
deviate from the official SemVer specification. Depending on the ecosystem you
are working with, the same semantic version string may be treated/interpreted
differently: for example both Maven and pip/PyPI treat versions 1.2.3.SP
differently because pip/PyPI lacks the notion of an SP post release. Apart
from that, 1.2.3.SP cannot be considered a valid semantic version according
to the SemVer spec.

Today we have a variety of different semantic versioning schemes:

• gem: gem requirement
• maven: Maven Dependency Version Requirement Specification
• npm: node-semver
• php: PHP Composer version constraints
• pypi: PEP440
• go: go semver
• nuget: NuGet semver
• conan: node-semver flavour

This SemVer versioning fragmentation limited the degree of automation we could apply to our
advisory extraction/generation process. This limitation motivated the
development of a methodology and tool semver_dialects that helps to digest and process semantic versions in a language agnostic way and, hence, helps to reduce the manual advisory curation effort.

Below, you can see an excerpt of the advisory information that is extracted and
generated by our semi-automated advisory generation process:

# ...
affected_range: ">=1.9,<=2.7.1||==2.8"
fixed_versions:
- "2.7.2"
- "2.8.1"
not_impacted: "All versions before 1.9, all versions after 2.7.1 before 2.8, all versions
  after 2.8"
solution: "Upgrade to versions 2.7.2, 2.8.1 or above."
# ...

In the excerpt above:

• affected_range denotes the range of affected versions which is the machine-readable, native syntax used by the package manager/registry (in this case pypi).
• fixed_versions denotes the concrete versions when the vulnerability has been fixed.
• not_impacted provides a textual description of the versions that are not affected.
• solution provides information about how to remediate the vulnerability.

To be able to extract and generate advisories like the one illustrated
above in a language/ecosystem agnostic way, we implemented and open-sourced a
generic semantic version representation and processing approach called
semver_dialects.

In the advisory excerpt above, the affected_range field contains the version
constraints in the native constraint syntax (in this case PyPI for Python);
fixed_versions can be inferred by inverting the affected_version (i.e.,
non-affected versions) and by selecting the first available version that falls
into the range of non-affected versions from the native package registry; this step
requires our approach to be able to parse the native semantic version syntax.

In order to deal with SemVer versioning and automatically process and generate the fields according to this
description, our semver_dialects implementation had to satisfy the following requirements:

1. Provide a unified interface to the language specific dialects.
2. Match semantic versions in a language agnostic way.
3. Invert ranges.
4. Cope with scattered, non-consecutive ranges.
5. Parse and produce different version syntaxes.
6. Parse and match versions/constraints in a best-effort manner.

SemVer versioning representation

First, we need a generic representation of a semantic version to start with. We
assume that a semantic version is composed of prefix and suffix where the
prefix contains segments for major, minor and patch version components as defined in the
SemVer specification. The suffix may hold additional information about pre/post
releases etc. As illustrated below, the major, minor and patch prefix segments
can be accessed by means of the corresponding methods.

s1 = SemanticVersion.new('1.2.3')
puts "segments: #{s1}"
# segments: 1:2:3
puts "major #{s1.major}"
# major 1
puts "minor #{s1.minor}"
# minor 2
puts "patch #{s1.patch}"
# patch 3

We cannot generally assume that all provided versions we would like to process
fully adhere to the SemVer spec which requires a version prefix (core) to
consist of three segments: major, minor and patch. Hence, per default, we
remove redundant, trailing zeros from the prefix to ensure that
2.0.0, 2.0 and 2 are considered identical.

Semver_dialects translates language specific version suffixes into numeric values. This process
can be described as version normalization. For example the Maven (pre-)release
candidate version 2.0.0.RC1 can be translated to a numeric representation
with prefix: 2 and suffix -1:1 by mapping RC to a numeric value (in this
example -1) and, thus, rendering it numerically comparable.

After this normalization step, semantic version matching for two versions vA
and vB can be implemented by simply numerically comparing their segments in a
pairwise fashion. For unknown suffices that are not mappable to the numeric
domain, we use lexical matching as a default fallback strategy.

In summary, comparing two semantic versions is a two-step process:

1. Normalization: Extend both semantic versions to have the same prefix length and suffix
   lengths by appending zeros.
2. Comparison: Iterate over segments and compare each of them numerically.

For example, after normalizing the versions 2.0.0.RC1 and 2.0.0 to 2:-1:1
and 2:0:0, respectively, we can iterate over the segments (delimited by
: in the example) which we can compare numerically to successfully identify
2:-1:1 as being the smaller (release-candidate) version in comparison to
2:0:0.

Constraint syntax - everything is a linear interval

Translating semantic versions into a generic representation makes them
numerically comparable which is already useful but not sufficient to express SemVer versioning constraints in a language-agnostic fashion.

For representing semantic version constraints in a generic way,
we rely on linear intervals. For the purpose of this blog, we define an interval as an ordered pair of two
semantic versions which we are referring to as lower and upper
bounds (or cuts). For the sake of simplicity, for the remainder of
this section we will use simple integers as examples for lower and upper bounds, respectively.

Linear intervals capture semantic version ranges symbolically which makes them
very versatile and space efficient. At the same time, we can rely on
well-established mathematical models borrowed from linear interval arithmetic
that enable us to translate/express any type of constraint in terms of
mathematical set operations on intervals.

In the table below you can find all the different types of intervals we
considered to model semantic version constraints and a corresponding
description where L stands for left, R stands for right with a and b
being the lower and upper bounds, respectively.

Type of interval	Example	Description
LR-closed	[a,b]: x >= a, x <= b	all versions starting from a until b
L-open R-closed	(a,b]: x > a, x <= b	all versions after a until b
L-closed R-open	[a,b): x >= a, x < b	all versions starting from a before b
LR-open	(a,b): x > a, x < b	all versions between a and b
L-unbounded	(-inf,b]: x <= b	all versions until b
R-unbounded	[a,+inf): x >= a	all versions starting from a

Below you can see example output for the different types of ranges from
semver_dialects where we are using the VersionParser component to generate
linear intervals from version constraints where , denotes a logical
conjunction: e.g., >=1, <=2 denotes the set of integers that are greater than or equal
to 1 and smaller than or equal to two, i.e., all integers/versions numbers starting from 1 until 2.

puts VersionParser.parse(">=1, <=2")
# [1,2]
puts VersionParser.parse(">1, <=2")
# (1,2]
puts VersionParser.parse(">=1, <2")
# [1,2)
puts VersionParser.parse(">1, <2")
# (1,2)
puts VersionParser.parse("<=2")
# (-inf,2]
puts VersionParser.parse(">=1")
# [1,+inf)

For solving SemVer versioning constraints, we use linear interval arithmetic
which is explained in-depth in the text-book "Introduction to Interval
Analysis."

As mentioned earlier, for our purposes, we define an interval as an ordered
pair of two semantic versions (lower and upper bound) that represents the set
of all those semantic versions that are enclosed by lower and upper bounds.
Given that intervals are sets, we can perform standard set operations on
them.

In the context of advisory generation, there are three operations we require to
satisfy all the requirements we defined earlier: Intersection, Union and Complement.
The operations are explained in more detail in the sections below.

For the remainder of this section, we explain interval operations, using two
example intervals X and Y with X=[x_l, x_u] and Y=[y_l, y_u] where
x_l, x_u denote the lower and upper bounds for X, and y_l, y_u denote
the lower and upper bounds for Y, respectively. In addition, we are using the
min and max functions, where max(a,b) returns the largest and min(a,b)
returns the smallest value of the parameters a and b; the ∅ symbol denotes
the empty set.

Intersection

The recipe below illustrates how the intersection (X ∩ Y) can be computed.

X ∩ Y = if X and Y have points in common [max(x_l,y_l), min(x_u,y_u)] else ∅

Intuitively, the intersection extracts the overlap (if any) from the two
intervals X and Y.

The code snippet below shows how the intersection is computed in semver_dialects for the two examples:

1. [2,5] ∩ [3,10]
2. [2,5] ∩ [7,10]

# 1. [2,5] ∩ [3,10] = [3, 5]
puts VersionParser.parse(">=2, <=5").intersect(VersionParser.parse(">=3, <=10"))
# [3,5]

# 2. [2,5] ∩ [7,10] = ∅
puts VersionParser.parse(">=2, <=5").intersect(VersionParser.parse(">=7, <=10"))
# empty

The intersection operation is useful to perform semantic version matching
for checking whether semantic version falls into a certain version interval
or range. For instance we may want to check whether version 1.2.3 satisfies
the constraint >=1.0.0, <1.2.4. In the context of Dependency Scanning, these types of
constraints are very common. The problem 1.2.3 ∈ [1.0.0, 1.2.4) can be
translated to a set intersection: [1.2.3, 1.2.3] ∩ [1.0.0, 1.2.4) =
[1.2.3, 1.2.3] which returns a non-empty set and, hence, tells us that
version 1.2.3 satisfies the given version constraints.

In the context of our advisory generation process, we use intersection to
cross-validate versions from vulnerability reports (CVEs) with versions of the
available package that are available on the package registry that serves it.

For convenience, as mentioned earlier, semver_dialects also supports grouping
intervals into ranges by means of the VersionRange class. A range is a set of intervals
which we denote with {I0, I1, ..., IN} where I denotes version intervals
delimited by , which can be interpreted as a union operator (explained in the next section).

A range is a set of intervals. In the example below, we first create a range
r1 to which we are adding two intervals: r1 = {[2.2.1, 5.1.2], (3.1, 10)}.
After that, there is a check for an overlap (i.e., an intersection) between
r1 and [0, 2.1) (no overlap) as well as [5.5, 5.5] (overlap). You can see
the output of semver_dialects in the excerpt below.

r1 = VersionRange.new
r1.add(VersionParser.parse(">=2.1.2, <=5.1.2"))
r1.add(VersionParser.parse(">3.1, <10"))

puts "[0,2.1) in #{r1}? #{r1.overlaps_with?(VersionParser.parse(">=0, <2.1"))}"
# [0,2.1) in [2.1.2,5.1.2],(3.1,10)? false
puts "[5.5,5.5] overlap with #{r1}? #{r1.overlaps_with?(VersionParser.parse("=5.5"))}"
# [5.5,5.5] overlap with [2.1.2,5.1.2],(3.1,10)? true

Union

The recipe below illustrates how the union (X ∪ Y) can be computed.

X ∪ Y = if X and Y have points in common {[min(x_l,y_l), max(x_u,y_u)]} else {X,Y}

The code snippet below shows how the union can be computed with
semver_dialects for the two examples:

1. [2,5] ∪ [3,10] = {[2,5], [3,10]} = {[2, 10]}
2. [2,5] ∪ [7,10] = {[2,5], [7,10]}

With the union operator, we can collapse version intervals in case they have an
overlap/intersection; otherwise, if X and Y are disjoint, we add their
intervals directly to the range.

# 1. [2,5] ∪ [3,10] = [2, 10]
puts "union: #{VersionParser.parse(">=2, <=5").union(VersionParser.parse(">=3, <=10"))}"
# union: [2,10]

# Version ranges perform union two for the purpose of automatically collapsing
# intervals (if possible)
r1 = VersionRange.new
r1.add(VersionParser.parse(">=2, <=5"))
r1.add(VersionParser.parse(">=3, <=10"))
puts "r1: #{r1}"
# union: [2,5],[3,10]
puts "r1 collapsed: #{r1.collapse}" # creates the union between intervals
# r1 collapsed: [2,10]

# 2. [2,5] ∪ [7,10] = {[2, 10], [7,10]}
r2 = VersionRange.new
r2.add(VersionParser.parse(">=2, <=5"))
r2.add(VersionParser.parse(">=7, <=10"))
puts "r2: #{r2}"
# r2: [2,5],[7,10]

In the context of Dependency Scanning, vulnerability data usually lists a set of intervals for
dependencies that are susceptible to a given vulnerability like the tensorflow example in the introduction where the following versions are affected:

1. up to 2.1.4
2. from 2.2.0 up to 2.2.3
3. from 2.3.0 up to 2.3.3
4. from 2.4.0 up to 2.4.2

This list of intervals can be represented as a single range (VersionRange) by
combining all of the mentioned version intervals through the union operator.

In the Ruby code example above, you can also see the collapse method which is
invoked on a VersionRange object. This method automatically collapses
overlapping intervals that are included in the same VersionRange to eliminate
redundant intervals. Collapsing the range {[2, 5], [3, 10]} yields a new range
{[2,10]} with only one interval while preserving semantic equivalence.

Complement

The recipe below, illustrates how the relative complement (X - Y) can be computed.

X - Y: Z := X ∩ Y;
if (z_l > x_l && z_u < x_u)
{[x_l, z_l),(z_u, x_u]}
else if (x_l < z_l)
{[x_l, z_l)}
else if (x_u > z_u)
{(z_u, x_u]}

Intuitively, this recipe computes the intersection (Z) between X and Y and
removes all elements from X that are included in the intersection. The
examples below illustrate the recipe:

1. [3, 5] - [1, 3]: with Z = [3, 3] we get {(3, 5]} which is
   equivalent to {[4, 5]}
2. [3, 10] - [10, 11]: with Z = [10, 10] we get {[3, 10)} which is equivalent to {[3, 9]}
3. [1, 5] - [2, 2]: with Z = [2, 2] we get [1, 2), (2, 5] which is equivalent to {[1, 1], [3, 5]}

With the recipe above, we can also compute the absolute complement X - Y by
assuming X is the universe that captures the entirety of all possible values:
(-inf,+inf). The universal complement can be defined as ~X = (-inf,+inf) - X.

With semver_dialects, the absolute complement can be computed by means of the
invert method as illustrated in the example below.

# example 1: ~[1,3] = {(-inf,0],[4, +inf)} = {(-inf,1),(3,+inf)}
r1 = VersionRange.new
r1.add(VersionParser.parse(">=1, <=3"))
puts r1.invert
# (-inf,1),(3,+inf)

# example 2: ~{[2.1.2, 5.1.2], (3.1, 10)} = ~{[2.1.2, 10)} = {(-inf,2.1.2),[10,+inf)}
{(-inf,0],[4, +inf)} = {(-inf,1),(3,+inf)}
r2 = VersionRange.new
r2.add(VersionParser.parse(">=2.1.2, <=5.1.2"))
r2.add(VersionParser.parse(">3.1, <10"))
puts r2.collapse.invert
# (-inf,2.1.2),[10,+inf)

In the context of Dependency Scanning, this functionality is used to automatically infer
non-affected versions from the affected versions information: if [1, 3]
represents all the affected versions of a vulnerable package, its complement
{(-inf,1),(3,+inf)}, per definition, captures only the unaffected version. In
our advisory generation process we cross-validate the version information of
packages from the package registries with this information about unaffected versions to check whether or not unaffected packages are available; if this is the case, we add the corresponding remediation information to the generated advisories.

Version Translation

Linear interval arithmetic provides us with all the means necessary to
represent and solve SemVer versioning constraints in a language-agnostic way.
However, in order to leverage the generic representation, we have to be able to
automatically translate the native semantic version dialects into the generic
representation and vice versa. The details of this translation functionality
are provided below.

Semver_dialects offers a VersionTranslator class. The VersionTranslator takes a native semantic version constraint, and translates
it into an intermediate string representation that can then be translated into a range (VersionRange) by using the VersionParser. Currently semver_dialects supports all the syntax listed below by invoking
translate_<package_type> where <package_type> is one of:

• gem: gem requirement
• maven: Maven Dependency Version Requirement Specification
• npm: node-semver
• packagist: PHP Composer version constraints
• pypi: PEP440
• go: go semver
• nuget: NuGet semver
• conan: node-semver flavour

The example below illustrates how the semver_dialects' VersionTranslator can
be used to translate native version syntax to an intermediate representation.
The VersionTranslator parses the native version syntax and translates it into
a common format. In the example below, you can further see that both
native, semantically equivalent but syntactically different version strings for
packagist and maven are translated into a common format: a string array
where a single array entry represents a conjunct of the semantic version
constraints. This translation step removes all language-specific features
from the native semantic version constraints.

# native packagist version constraint syntax
vs_packagist = "<2.5.9||>=2.6.0,<2.6.11"
# native maven version constraint syntax
vs_maven = "(,2.5.9),[2.6.0,2.6.11)"

# translate
puts VersionTranslator.translate_packagist(vs_packagist).to_s
# ["<2.5.9", ">=2.6.0 <2.6.11"]
puts VersionTranslator.translate_maven(vs_maven).to_s
# ["<2.5.9", ">=2.6.0 <2.6.11"]

This common format can then be translated to a version interval by means of
VersionParser and VersionRange. The example below illustrates how the
version interval constraint is generated by iterating over the array elements
of our intermediate representation, translating them to intervals and adding
these intervals to the VersionRange object constraint. At the end of the
excerpt below, we check whether version 1.0.0 satisfies the version
constraint <2.5.9||>=2.6.0,<2.6.11 which correctly yields true.

# translate native maven version constraint to range of interval
constraint = VersionRange.new
VersionTranslator.translate_maven(vs_maven).each do |version_string|
  constraint << VersionParser.parse(version_string)
end

puts constraint.overlaps_with?(VersionParser.parse('=' + '1.0.0'))
# true

Wrapping it up

We discussed the fragmentation of SemVer versioning which poses a challenge
when building automation around semantic version processing for
multi-language/ecosystem applications. In this blog post, we used our internal
semi-automated process for advisory generation as an example.

We illustrated how we tackled the above-mentioned challenge by building a
generic/language-agnostic semantic version approach based on linear interval
arithmetic. All mechanisms discussed in this blog post are implemented in the open-sourced (MIT)
semver_dialects implementation and published on rubygems.org.