Quickstart

Follow the steps below to start syncing data from CloudQuery source plugins (opens in a new tab) to destination plugins (opens in a new tab).

Download CloudQuery CLI

You can download the precompiled binary from releases (opens in a new tab), or using CLI:

brew install cloudquery/tap/cloudquery

Sign in with CloudQuery

We recommend that you authenticate using the CloudQuery CLI. Once signed in, you can use CloudQuery with premium plugins up to a free quota and premium plugins in preview without limitations.

Authenticated users also have higher rate limits on plugin downloads than anonymous users.

You will be able to create a new account when you sign in with the CLI for the first time. Alternatively, head to CloudQuery Cloud (opens in a new tab) to create a new account.

To sign in from the CLI, run the following command:

cloudquery login

A new browser window will open where you will complete the sign-in process.

For a non-interactive sign-in process and sync, you can generate an API key. This is useful for production deployments.

Configure Destination Plugin

Destination plugins define where you will be syncing your data to (See: available destinations (opens in a new tab)).

For example, let's configure the PostgreSQL (opens in a new tab) destination plugin (See all available versions here (opens in a new tab)). Create a file in a new directory cloudquery-config/postgresql.yml:

kind: destination
spec:
  ## Required. name of the plugin.
  ## This is an alias so it should be unique if you have a number of postgresql destination plugins.
  name: "postgresql"

  ## Optional. Where to search for the plugin. Default: "github". Options: "github", "cloudquery", "local", "grpc".
  # registry: "github"

  ## Path for the plugin.
  ## If registry is "github" path should be "repo/name"
  ## If registry is "cloudquery" path should be "team/name"
  ## If registry is "local", path is path to binary. If "grpc" then it should be address of the plugin (usually useful in debug).
  path: "cloudquery/postgresql"

  ## Optional. Registry to download the plugin from. Defaults to "github"
  registry: "cloudquery"

  ## Required. Must be a specific version starting with v, e.g. v1.2.3
  ## checkout latest versions here https://github.com/cloudquery/cloudquery/releases?q=plugins-destination-postgresql&expanded=true
  version: "v8.2.6"

  ## Optional. Default: "overwrite-delete-stale". Available: "overwrite-delete-stale", "overwrite", "append". 
  ##  Not all modes are supported by all plugins, so make sure to check the plugin documentation for more details.
  write_mode: "overwrite-delete-stale" # overwrite-delete-stale, overwrite, append

  spec:
    ## plugin-specific configuration for PostgreSQL.
    ## See all available options here: https://github.com/cloudquery/cloudquery/tree/main/plugins/destination/postgresql#postgresql-spec

    ## Required. Connection string to your PostgreSQL instance
    ## In production it is highly recommended to use environment variable expansion
    ## connection_string: ${PG_CONNECTION_STRING}
    connection_string: "postgresql://postgres:pass@localhost:5432/postgres?sslmode=disable"

All general options for destination spec you can find under references/destination-spec.
All options for postgresql destination plugin spec you can find here (opens in a new tab)

For a list of all available destination plugin spec options, see Destination Spec Reference.

💡

If you have multiple destination plugins (or running multiple cloudquery instances in parallel), it's required that every plugin configuration has a unique name (e.g. postgres1, postgres2, …).

Configure Source Plugin

Source plugins define the APIs you want to sync from (See available sources (opens in a new tab)).

For example, let's configure the AWS (opens in a new tab) source plugin. Create an aws.yml file in your cloudquery-config directory:

kind: source
spec:
  ## Required. name of the plugin to use.
  ## This should be unique if you have number of aws plugins.
  name: "aws"
  ## Optional. Where to search for the plugin. Default: "github". Options: "github", "cloudquery", "local", "grpc"
  # registry: "github"

  ## Path for the plugin.
  ## If registry is "github" path should be "repo/name"
  ## If registry is "cloudquery" path should be "team/name"
  ## If registry is "local", path is path to binary. If "grpc" then it should be address of the plugin (usually useful in debug).
  path: "cloudquery/aws"

  ## Optional. Registry to download the plugin from. Defaults to "github"
  registry: "cloudquery"

  ## Required. Must be a specific version starting with v, e.g. v1.2.3
  ## checkout latest versions here https://github.com/cloudquery/cloudquery/releases?q=plugins-source-aws&expanded=true
  version: "v27.8.0"

  ## Required. You can use ["*"] to sync all tables or specify specific tables. Please note that syncing all tables can be slow
  ## See all tables: https://www.cloudquery.io/docs/plugins/sources/aws/tables
  tables: ["aws_s3_buckets"]

  ## Required. all destinations you want to sync data to.
  destinations: ["postgresql"]

  spec:
    ## Optional. plugin specific configuration
    ## By default will use the current aws credentials available (just like AWS CLI)
    ## See all available options here: https://github.com/cloudquery/cloudquery/blob/main/plugins/source/aws/docs/configuration.md

All general options for source spec you can find under references/source-spec.
All options for aws source plugin spec you can find here (opens in a new tab)

💡

If you have multiple source plugins (or running multiple cloudquery instances in parallel), it's required that every plugin configuration has a unique name (e.g. aws1, aws2, …).

Running multiple plugins with the same name can cause unexpected behavior (e.g. empty tables).

You can read more here.

Authenticate with the Cloud Provider

CloudQuery uses similar authentication methods to the official SDKs of each cloud provider. The AWS authentication guide is located here (opens in a new tab).

You can find authentication for other plugins in their corresponding READMEs (opens in a new tab).

Start Syncing

Now you can start syncing the data from your source plugins to the specified destinations.

You can run the sync by specifying the directory with the configuration files or each file individually

cloudquery sync ./cloudquery-config

# or cloudquery sync cloudquery-config/aws.yml cloudquery-config/postgresql.yml

You should see a spinner with number of resources synced and the time it took to sync. Once the sync is done you can query the data via the database directly.

Note: All logs are written by default to cloudquery.log so you can easily see if any APIs failed and analyse those further.

Introduction Linux