Architecture
This is an advanced section describing the inner workings and design of CloudQuery. (It might be useful when developing new plugins).
CloudQuery has a pluggable architecture and uses the gRPC (opens in a new tab) to communicate between plugins.
CloudQuery CLI Responsibilities
- Main entry point and CLI for the user.
- Reading CloudQuery configuration.
- Downloading, verifying, and running sync from source to destination plugins
CloudQuery Plugin Responsibilities
- Intended to be run only by CloudQuery CLI.
- Communicates with CloudQuery CLI over gRPC to receive commands and actions.
- Source Plugins: Initialization, authentication, and fetching data via third-party cloud/SaaS API.
- Destination Plugins: Authentication, Database migrations, Data Insertion.
SDK
CloudQuery plugins utilize plugin-sdk, which abstracts most of the TL (in ETL, extract-transform-load). So, as a developer, you will only have to implement the ("E" in "ETL") initializing, authentication, and fetching of the data via the third-party APIs — the SDK will take care of transforming the data and loading it into the database. Also, your plugin will get support out-of-the-box for new features and things like other database support as cloudquery-core progresses.