Feature Update: Vulnerability Query Builders

NopSec is proud to announce the release of a new feature this week aimed at enhancing our users’ workflows! We met with many of our users and worked to understand where they were facing pain points during their workflows. The end result of this latest addition is a new and improved querying system for Unified VRM.

Customer Requested Update Drivers 

The following were the main drivers we aimed at solving with the new Query Builder.

• Logical Operators: All filters in our existing query language were assumed to be connected with an “AND”, even though it was not shown. Experienced users often asked if they could make a query using “OR” or “NOT” operators. Previously, we couldn’t support those requests.
• User Experience: The general workflow using our existing query builder was relatively easy and favored the Graphical User Interface version over the Text Based version. However, there were a handful of experience issues, such as: 
  • Being unable to edit filters once they were set or 
  • Having to specify dates manually without a date picker or b
  • Being unable to use relative dates 

This often caused users to turn to the text based editor over time.

• Filters: In our existing query language many of the filters available were specific to infrastructure based assets and there were not many filters available for the other asset types. There were also vulnerability specific filters that were not being shown or available that made filtering for specific vulnerabilities difficult at best or impossible at worst.

New Query Builder Features

Below you’ll find a quick overview list of all the new features that we’ll go over in this post related to our updated query builder:

• New query language which includes “AND”, “OR”, “NOT” logical operators
• Text based editor by default
• New GUI query builder interface
• Date experience improvements with date pickers and relative time support
• Over 30 new filters
• General user experience improvements

Release Timing

Starting in mid-May we will do the initial release of UVRM 5.5 which will include the new query for users to choose when creating reports. You’ll be able to do so by clicking the “New Report V2” button.

Later this summer we’ll be releasing UVRM 6.0. In this version update, the new query builder will become the default and only option for all users to select.

Data Access

It’s important to note up front that all queries adhere to each user’s data access controls. This means users will only be able to see data they have been given access to see regardless of the query they provide. This applies to type-ahead search recommendations, drop-downs, and report table data.

Text vs GUI Query Builders

The first thing you’ll notice when using the new query builder is that the default will be the text based query builder over a GUI based one.

A few reasons for this. First, one of the core pain points users faced was based on the ability to quickly edit a query. The text based version allows for users to quickly modify their query without having to click any buttons or switch context. This includes the ability to quickly add a new filter as well. We understand that the learning curve may be high at first. To access the GUI query builder click the icon (pictured below) on the far right of the text input window.

This will open the GUI query builder.

Over time, we hope that the query syntax and the way we’ve designed the filters will be easy to learn and hope that you’ll use the text editor natively. That said, we’ll continue improving the GUI query builder as well as there are several use cases that will only work in that space.

Text Based Query Builder Query Language

We worked hard to find a query system that was used across many different applications and use cases and landed with Google’s Common Expression Language (CEL) as the main compiler for our query system. 

This means that our queries are performant and support some pretty complex logical operators out of the box. We are aiming to make our query syntax straightforward and simple with filters using a common format to quickly find and remember so that you can quickly learn to use the query builder without any UI help. Below is how our query language is configured.

The format is generally the following:

• A query consists of one or more “groups” as represented by the parentheses “(“ and “)”.
• Each “group” contains one or more “rules.” The above graphic only shows one rule per group.
• Each rule consists of a filter (category +”.”+ filter) and a value, joined by a filter operator. Shown here as a “=” to represent equals but could be =, <, <=, >, >=, ==, in. All depending on the specific filter and data type.
• Rules or groups can be joined by a logical operator; “OR”, “AND”.
• Rules or groups can be preceded by a “NOT” to exclude the specific rule or group from the results such as “AND NOT” or “OR NOT”.

The above query can be condensed down in the following ways:

Here the query can be condensed down to one rule because the “ORs” were compared against the same filter; asset.ip. So the values become a List of values similar to many programming languages consisting of bookended brackets and values in quotes separated by commas. Ex. [“windows”, “mac”, “unix”]

The above query can be further condensed down by removing the bookending parentheses and changing the query from an “IN” operator to an “=” operator.

An internal rule “OR” is supported by placing a comma in the string, however, if the string you’re looking for contains a “,” then it is best to use the bookending brackets instead.

Example queries:

• vuln.status in [“OPEN”] and asset.ip = “1.2.3.4”
• vuln.status in [“OPEN”] and asset.ip = “1.2.3.4” and vuln.instance_grade in “D”
• vuln.status in [“OPEN”] and asset.ip = “1.2.3.4” and vuln.instance_grade in “D” and threat.internet_facing
• vuln.status in [“OPEN”] and asset.ip = “1.2.3.4, 2.3.4.5” and vuln.instance_grade in “D” and threat.internet_facing
• vuln.status in [“OPEN”] and (asset.ip = “1.2.3.4, 2.3.4.5” or asset.hostname = “super”) and vuln.instance_grade in “D” and threat.internet_facing
• vuln.status in [“OPEN”] and not (asset.ip = “1.2.3.4, 2.3.4.5” or asset.hostname = “super”) and vuln.instance_grade in “D” and threat.internet_facing

Text Based Query Builder Data Type Filters

One of the key features UVRM provides clients is the ability to aggregate VM data from across all of their scanners. In order to operationalize this data users need the ability to efficiently query for it or filter it out to create reports and remediation plans. 

In order to support users ability to learn the text based query system and provide some organization we are leveraging filter categories. New Categories will be:

• Application = Filters related to SAST and DAST asset metadata only will be placed here.
  • Ex. app.branch
• Artifact = Filters related to asset metadata typically related to Containers, Images, Packages will fall under Artifacts.
  • Ex. artifact.running
• Asset = Filters related to asset metadata that are not specific to Applications, Cloud assets, or Artifacts will go here.
  • Ex. asset.type
• Cloud = Filters related to AWS, Azure (coming soon), Google (coming soon) will be here.
  • Ex. aws.zone
• CMDB = Filters related to data brought in from your CMDB and associated with one of your assets will live here.
  • Ex. cmdb.ip
• Scanner = Filters focused on allowing you to filter results based on a particular Scanner. In the future this may include other scanner specific filters.
  • Ex. scanner
• Tags = Filters focused on Asset Tag Groups.
  • Ex. tags.name
• Threat = Filters based on threat insights will be placed here.
  • Ex. threat.internet_facing
• Vuln = Filters based on vulnerability data will go here.
  • Ex. vuln.title

You can download the Query Syntax Cheat Sheet here.

Leveraging the GUI Query Builder

To open up the GUI helper you click on the icon located in the far right to open the drawer.

You’ll now see our new GUI based query builder.

You should notice that many of the components from the text based editor format are now visible in the GUI.

• You can create one or more Rules by clicking on the Add Rule button.
• You can create one or more Groups by clicking on the Add Inner Group button to create a new group. You can then drag and drop existing rules into that group or you can add new rules by clicking the Add New Rule button within the Group.
• You can reorder any of the rules or groups as you like by clicking and dragging them.
• You can set the logical operator by changing the AND or OR field for each group.
• You can set a specific rule or group as a NOT rule by clicking the NOT checkbox. 
• You’ll notice that anything you change or add in the GUI version is also reflected in the text editor. This allows you to learn the language over time and allows you to quickly edit a filter within the text editor without having to open the GUI helper.

GUI Query Builder User Experience Improvements

Today, when using the GUI query builder many of the filters support type-ahead search to help you discover your own data and find specific items you’re looking for. A great user experience improvement we’ve implemented is to filter down your type-ahead suggestions to the data already filtered by your query rules. This means that as you add new filters you will be presented with specific options to deep dive further into your dataset. You can override this by typing your own value and selecting the “Search by …” option in the dropdown menu. We’ll support type-ahead support in the text based query system in the future.

Editing queries is much simpler than before. In the old query system you had to delete the filter, add a new filter, find your filter, and re-configure the values if you had made a mistake and wanted to edit the filter. Today, you’ll be able to directly edit the values directly in the text based query editor or in the GUI query builder within two clicks or less.

Date pickers are something that is normal in any query system and now they are available in the GUI query builder whenever the filter is a Date based filter. You can override this by typing the value as well.

Relative time support was one of the most requested features for reports as there was no way of configuring a report to use relative times. Today this is supported in the text based query editor and it will be supported in the GUI query builder in the next iteration. See the Cheat Sheet for examples but here’s a simple example; vuln.detected_date > “-6m” which translates to vuln.detected_date > 12/15/2023.

What’s Coming Next?

For the query system we will continue releasing new filters to continue to support complex querying to build highly specific reports to meet your needs. We will also listen to your feedback and provide improvements to syntax, categories, and usage in order to provide user experience improvements such as better type-ahead support in the text based query editor or in the GUI query builder.