An Introduction to OWASP Amass 4 - Part 3 - The Database
The OWASP Amass project is an open-source, actively developed security tool with extensive community support that focuses on information gathering and reconnaissance. It helps security researchers and penetration testers discover and map the attack surface of their target networks by using a variety of data sources. Whether you are a penetration tester, an auditor, a security researcher or the CISO/IT manager, you have several valid reasons for mapping out the external attack surface of an organisation. This process is also referred to as reconnaissance or information gathering.
Version 4 is a major revision of Amass. If you are familiar with earlier versions then you will need to change your approach to understand how it is organized and how this "framework" works.
In this instalment in our series on OWASP Amass version 4 architecture lets focus on the database that stores assets the Amass command line tool component discovers. This is part 3 of the series. Part 1, is an introduction to the Amass GitHub, Part 2 discusses the data model and the approach to configuration in your workflow.
The Database
Amass needs to store what it finds. Other Amass tools in the ecosystem need to be able to read and process the assets discovered and stored. In Amass prior to version 4 used an SQLite database, and you could specify which SQLite file on the command line. SQLite is still supported in version 4, however support for a Postgres database has been added. Your decision to choose SQLite versus Postgres will depend on your workflow and project management.
Asset Organization
Regardless of which approach you choose (SQLite or Postgres) the compartmentalization of a targets assets is recommended. What we mean by this is that if you are working with different clients (either simultaneously or not) separating a client’s reconnaissance findings into different database is a best practice. This separation will prevent any accidental data exposure between clients. By maintaining a strong compartmentalization in different databases or SQLite file help avoid accidental sharing of data between clients at report time.
Database Account
It is important that we spend some time discussing account access to the database. Depending on your environment, your risk appetite, and if you want to follow best practices, its best not to access the database as superuser, or as db-admin. The asset-db user guide provides instructions for this use case. However in our isolated example environment we will use db-admin. A later blog will cover this use case separately.
The asset-db Repository
The repository for the asset-db `database interaction layer` resides at https://github.com/owasp-amass/asset-db. Within there will be documentation in the *docs* folder.
We could install Postgres on our linux host, but for isolation and ease of deployment I chose containers. Yes, you will need to have docker installed as well as docker-compose. Both of these installs are out of scope for this guide. Docker provides documentation for installing Docker and installing docker-compose on different platforms.
At a high level, setting up the database tier consists of:
Clone the asset-db repository.
Modify the docker-compose file to suit.
Configure the .env.local to suit.
Launch postgres container.
Create the Amass Database.
Set the database timezone.
Connecting to the Database.
Install and Confirm pg_trgm Extension Status.
Fortunately the asset-db repository contains a docker-compose and .env.local files we can use.
We will also need to perform some specific configuration on the database file we will use as shown in the steps below.
Step 1: Clone the asset-db repository
└─$ cd /opt
┌──(user㉿kanga)-[/opt]
└─$ git clone https://github.com/owasp-amass/asset-db.git
Cloning into 'asset-db'...
remote: Enumerating objects: 319, done.
remote: Counting objects: 100% (143/143), done.
remote: Compressing objects: 100% (87/87), done.
remote: Total 319 (delta 74), reused 90 (delta 52), pack-reused 176
Receiving objects: 100% (319/319), 115.63 KiB | 2.27 MiB/s, done.
Resolving deltas: 100% (164/164), done.
┌──(user㉿kanga)-[/opt]
└─$ cd asset-db
┌──(user㉿kanga)-[/opt/asset-db]
└─$ ls -la
total 120
drwxrwxr-x 8 user user 4096 Jun 28 10:38 .
drwxrwxrwx 11 root root 4096 Jun 28 10:38 ..
-rw-rw-r-- 1 user user 5071 Jun 28 10:38 assetdb.go
-rw-rw-r-- 1 user user 20508 Jun 28 10:38 assetdb_test.go
-rw-rw-r-- 1 user user 219 Jun 28 10:38 docker-compose.yml
-rw-rw-r-- 1 user user 307 Jun 28 10:38 Dockerfile
drwxrwxr-x 2 user user 4096 Jun 28 10:38 docs
-rw-rw-r-- 1 user user 148 Jun 28 10:38 .env.local
drwxrwxr-x 8 user user 4096 Jun 28 10:38 .git
drwxrwxr-x 3 user user 4096 Jun 28 10:38 .github
-rw-rw-r-- 1 user user 478 Jun 28 10:38 .gitignore
-rw-rw-r-- 1 user user 1636 Jun 28 10:38 go.mod
-rw-rw-r-- 1 user user 9297 Jun 28 10:38 go.sum
-rw-rw-r-- 1 user user 4365 Jun 28 10:38 initdb-assetdb.sh
-rw-rw-r-- 1 user user 11357 Jun 28 10:38 LICENSE
-rw-rw-r-- 1 user user 116 Jun 28 10:38 Makefile
drwxrwxr-x 4 user user 4096 Jun 28 10:38 migrations
drwxrwxr-x 2 user user 4096 Jun 28 10:38 repository
drwxrwxr-x 2 user user 4096 Jun 28 10:38 types
Note the docker-compose.yml file and the .env.local files. The docker-compose.yml file contains the docker instructions for creating the container, references the .env.local file for the Postgres environment, and sets admin user and password for the database.
Step 2: Modify the docker-compose.yml to Suit
Modify docker compose file if required. Here, we will only expose the database port on a local address. I do not want this port to be viewable from outside my system so I modified ports as below:
version: '3'
services:
postgres:
container_name: assetdb_postgres
image: postgres:latest
restart: always
env_file: .env.local
ports:
- "127.0.0.1:5432:5432"
volumes:
postgres-db:
driver: local
Step 3: Modify .env.local to Suit
Lets have a look at the .env.local file in the cloned directory
┌──(user㉿kanga)-[/opt/asset-db]
└─$ cat .env.local
POSTGRES_USER=postgres
POSTGRES_PASSWORD=<some-password>
POSTGRES_DB=postgres
SQLITE3_DB=test.db
AMASS_DB=example
AMASS_USER=amass
AMASS_PASSWORD=amassTo start we are only interested in the POSTGRES settings. Modify .env.local to change admin user and password.
POSTGRES_USER=<database-admin-name>
POSTGRES_PASSWORD=<some-password>
POSTGRES_DB=postgresThe database name in this example has been left as “postgres”. As the workflow evolves, we could define different environment files for different projects and use different database name. If we wish to add databases later to the same database server for different project then we would need to ensure the `“pg_trgm”` extension is available for the database(more on that later). Of course, enter your postgres username and password details in the space provided.
Step 4: Launch Postgres Container
In the same directory as the docker-compose and .env.local files run docker-compose. You will need to have privileges to run docker-compose.
└─$ sudo docker-compose up -d
[sudo] password for user:
Creating network "asset-db_default" with the default driver
Creating volume "asset-db_postgres-db" with local driver
Pulling postgres (postgres:latest)...
latest: Pulling from library/postgres
2cc3ae149d28: Already exists
d1a63825d58e: Pull complete
----------------------8<----------------------------------------
6a489170d05e: Pull complete
440b39aff272: Pull complete
582c79113570: Pull complete
Digest: sha256:46aa2ee5d664b275f05d1a963b30fff60fb422b4b594d509765c42db46d48881
Status: Downloaded newer image for postgres:latest
Creating assetdb_postgres ... doneAnd now we can confirm that the container is running:
┌──(garth㉿kanga)-[/opt/asset-db]
└─$ sudo docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
15aa2854662c postgres:latest "docker-entryp" 9 minutes ago Up 8 minutes 127.0.0.1:5432->5432/tcp assetdb_postgres
Step 5: Create the Amass Database
Now that we have a running database server we need to create a database (bag of holding) for Amass to store data in. In the following cases we use a tool called **PSQL** and following steps from the asset-db user guide we create the database for our project. But first, in case your distro (ubuntu?) does not have the PSQL command line tool installed lets install it.
From the command line:
┌──(user㉿kanga)-[/opt/asset-db]
└─$ sudo apt-get install postgresql-client
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
libpq5 postgresql-client-12 postgresql-client-common
Suggested packages:
postgresql-12 postgresql-doc-12
The following NEW packages will be installed:
libpq5 postgresql-client postgresql-client-12 postgresql-client-common
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 1,202 kB of archives.
After this operation, 4,477 kB of additional disk space will be used.
Do you want to continue? [Y/n] yNow we can create our database:
psql postgres://<database-admin-name>:<some-password>@127.0.0.1:5432 -c "CREATE DATABASE assetdb"Of course you need to enter the database username and password that you specified in .env.local in step 3 above.
**Note**:
Here we use the name `assetdb` as used in the User Guide. But as we mentioned above we have many targets and therefore as a best practice we suggest using different databases for different clients. Therefore, we need to follow these steps for each database we create.
After the database is created we should see the following:
CREATE DATABASEStep 6: Set the Database Timezone
And as per the user guide we set a timezone on that database.
psql postgres://<database-admin-name>:<some-password>@127.0.0.1:5432 -c "ALTER DATABASE assetdb SET TIMEZONE to 'UTC'"Which should respond with:
ALTER DATABASEStep 7: Connect to the Database
With the database created and the timezone set, we connect to the database server to query and modify settings as well as add any necessary extensions.
**Note**:
Here we need to ensure that Amass can access the `pg_trgm` extension. To do this we must specify the database Amass is to connect to when we connect with psql. If we do not specify the intended database then we will not be able to query to see enabled extensions for that database. In this example we will specify "assetdb", the database we created above. Refer to the issue at `Panic from Containerized Postgresql` in the troubleshooting channel of the Amass Discord to see what errors occur when the `pg_trgm` extension is not enabled.
┌──(user㉿kanga)-[/opt/asset-db]
└─$ psql "host=127.0.0.1 port=5432 user=<database-admin-name> password=<some-password> dbname=assetdb"
psql (12.18 (Ubuntu 12.18-0ubuntu0.20.04.1), server 16.3 (Debian 16.3-1.pgdg120+1))
WARNING: psql major version 12, server major version 16.
Some psql features might not work.
Type "help" for help.If we do not explicitly specify the database in the `dbname` parameter, then the commands I am about to execute will not apply to our target database and Amass will encounter issues. Note that my prompt “assetdb” implies that we are connected to that specific database on our database server.
Step 8: Install and Confirm pg-trgm Extension and Status
The pg_trgm extension improves the performance of queries that use the LIKE operator. This section determines if it is installed and if not, installs it.
We want to determine if the “pg_trgm” extension is enabled for our database. To list the installed extensions in postgres use `\dx` as described in psql notes. The code snippet below shows the command and its result.
assetdb=# \dx
List of installed extensions
Name | Version | Schema | Description
---------+---------+------------+------------------------------
plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language
(1 row)The result above tells me that `pg_trgm` is not installed. I could also execute:
assetdb=# SELECT * FROM pg_extension where extname = 'pg_trgm';
oid | extname | extowner | extnamespace | extrelocatable | extversion | extconfig | extcondition
-----+---------+----------+--------------+----------------+------------+-----------+--------------
(0 rows)Clearly we need to install the extension. How I go about it depends on if I am going to run this as admin or as another less privileged user. In this example I am going to the postgres server admin.
assetdb=# CREATE EXTENSION pg_trgm SCHEMA public;Now when I query the server I will see that pg_trgm is installed.
assetdb=# SELECT * FROM pg_extension where extname = 'pg_trgm';
oid | extname | extowner | extnamespace | extrelocatable | extversion | extconfig | extcondition
-------+---------+----------+--------------+----------------+------------+-----------+--------------
16389 | pg_trgm | 10 | 2200 | t | 1.6 | |
(1 row)and
assetdb=# \dx
List of installed extensions
Name | Version | Schema | Description
---------+---------+------------+-------------------------------------------------------------------
pg_trgm | 1.6 | public | text similarity measurement and index searching based on trigrams
plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural languageWith the pg_tgrm extension installed and privileges set, we can exit psql.
assetdb=# \qA Note on Privileged Access
So far we have created and accessed the DB using a privileged database account. However, as a best practice outlined in the asset-db user guide we should not access the database with a superuser account. Therefore we would want to create a non-superuser database account for database access. The user guide provides instructions on how to do this clearly and therefore we will not repeat them here.
Wrap Up
This section focused on setting up a PostgreSQL database server using a docker container and creating a database with the appropriate extension.
Next, we will focus on the installation of the Amass Command Line Interface (CLI) tool.
