Overview

The GCP Configuration View enables users to securely connect the Cavisson platform to their Google Cloud Platform (GCP) environment. This view supports multiple authentication methods using service accounts and JWT configurations, and allows setup of Cloud Cost Visibility for analyzing GCP billing data. The integration supports JSON Web Tokens (JWT) and GCE default service account credentials for authentication and billing export access.

Prerequisites

Google Cloud Billing Account: Active billing account to track and manage GCP costs.
Google Cloud Project (linked to billing): Project linked with the billing account for cost data export.
Google Cloud Storage (GCS) Bucket: Required as the storage destination for the exported billing CSV files. Create a bucket in Cloud Console → Storage → Buckets → Create with a unique name (e.g., gcp-daily-cur), choose location (Region or Multi-region), storage class (Standard), and click Create.
Billing Export to BigQuery Required to automatically collect detailed daily/hourly billing data from GCP. In Cloud Console → Billing → Settings → Billing Export, enable BigQuery Export, select the target project, create a dataset (e.g., billing_dataset), and enable Detailed usage cost export to receive billing data in BigQuery.
SQL for Daily CSV Export: Required to transform and export processed billing data from BigQuery into CSV files for analysis. Write and execute a BigQuery SQL query to export daily processed billing data in CSV format to the GCS bucket, using an EXPORT DATA OPTIONS statement with compression (GZIP) and proper file naming.
BigQuery Dataset for Processed Data: Required to store the transformed cost and usage data for reporting. Create a new dataset in BigQuery (e.g., cur_export_dataset) to store processed cost and usage data.

Example Query:


DECLARE year_month_day STRING DEFAULT FORMAT_DATE('%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY));

DECLARE billing_period STRING DEFAULT FORMAT_TIMESTAMP('%Y-%m', CURRENT_TIMESTAMP());

DECLARE export_uri STRING;



SET export_uri = CONCAT(

  'gs://<bucket_name>/<prefix>/BILLING_PERIOD=',

  billing_period,

  '/billing-flat-',

  year_month_day,

  '-*.csv.gz'

);



EXPORT DATA OPTIONS (

  uri=(export_uri),

  format='CSV',

  compression='GZIP',

  header=TRUE,

  overwrite=TRUE

) AS

SELECT

  billing_account_id,

  service.id AS service_id,

  service.description AS service_description,

  sku.id AS sku_id,

  sku.description AS sku_description,

  usage_start_time,

  usage_end_time,

  project.id AS project_id,

  project.number AS project_number,

  project.name AS project_name,

  project.ancestry_numbers AS project_ancestor_number,

  location.location,

  location.country,

  location.region,

  resource.name as resource_name,

  resource.global_name as resource_global_name,

  location.zone,

  transaction_type,

  seller_name,

  export_time,

  cost,

  currency,

  currency_conversion_rate,

  usage.amount AS usage_amount,

  usage.unit AS usage_unit,

  usage.amount_in_pricing_units,

  usage.pricing_unit,

  invoice.month AS invoice_month,

  invoice.publisher_type AS invoice_publisher_type,

  cost_type,

  adjustment_info.id AS adjustment_id,

  adjustment_info.description AS adjustment_description,

  adjustment_info.mode AS adjustment_mode,

  adjustment_info.type AS adjustment_type,

  cost_at_list

FROM `<project_id.datsetname.datasettablename>`

WHERE DATE(usage_start_time) = DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY);

GCP Configuration

Name: Provide a unique name to identify this GCP configuration in the platform.

Connection Details: Choose an authentication method to connect to GCP. Google JWT File: Authenticate using a JWT service account key.

Upload JWT File: Upload the .json key file containing service account credentials.
Retrieve via: Go to GCP Console > IAM & Admin > Service Accounts > Select Service Account > Keys > Add Key > Create new key > JSON format > Download.
Paste JWT File: Paste entire JSON content into input area.
Fill Details of JWT Manually: Manually enter the following fields extracted from the JWT .json:
- Project ID: Identifier of your GCP project.
  To retrieve: GCP Console > Home > Select Project > View Project ID.
- Client Email: Email associated with the service account.
  To retrieve: Go to IAM & Admin > Service Accounts > Select Account.
- Token URI: The token endpoint used to obtain OAuth 2.0 access tokens. Usually: https://oauth2.googleapis.com/token
- Private Key: The private key string used for authentication. (Keep this secure.)

GCE Default Service Account: Alternatively, users can allow the platform to use the default credentials available on GCE (Google Compute Engine) VMs.

Project ID: GCP project ID where the instance is hosted.
To retrieve: GCP Console > Home > Select Project.
Client Email: Default service account email assigned to the instance.
To retrieve: GCP Console > IAM & Admin > Service Accounts.
Token URI: The OAuth 2.0 token endpoint
Private Key: Auto-managed by GCP for default service account; not required to input manually.

Cloud Cost Visibility Configuration

After authentication, configure how to access billing and usage data for cost reporting and analytics.

Billing Data Dataset Name: Name of the BigQuery dataset that stores billing data.
To retrieve: GCP Console > Billing > Reports > BigQuery export.
Billing Data Table Name: Name of the table inside the dataset containing the cost and usage report.
To retrieve: GCP Console > BigQuery > View Dataset > Select Table.
Bucket Name:Name of the Cloud Storage bucket used (if applicable) for billing export storage.
To retrieve: GCP Console > Cloud Storage > Buckets.
Bucket Prefix: The folder path or prefix within the bucket where billing data is stored.

Test and Save: Validates credentials and configuration, then saves if successful.

Reset: Clears all fields for fresh input.

Configure the tenant

Make some preparations steps before connect a new GCP data source in Cavisson, go to the GCP cloud.

Prepare a role for Cavisson. There are two ways, select one:
- Run the following command in GCP CLI:
  gcloud iam roles create Cavisson_connection_role \--project=hystaxcom \--permissions=bigquery.jobs.create, bigquery.tables.getData, compute.addresses.list, compute.addresses.setLabels, compute.disks.list, compute.disks.setLabels, compute.instances.list, compute.instances.setLabels, compute.images.list, compute.images.setLabels, compute.machineTypes.list, compute.networks.list, compute.regions.list, compute.snapshots.list, compute.zones.list, iam.serviceAccounts.list, monitoring.timeSeries.list, storage.buckets.get, storage.buckets.list ...
- Via Google Cloud console
  1. Go to Roles page and click Create Role.
  2. Give the role any name and description.
  3. Add the following permissions:
    - bigquery.jobs.create
    - bigquery.tables.getData
    - compute.addresses.list
    - compute.addresses.setLabels
    - compute.disks.list
    - compute.disks.setLabels
    - compute.firewalls.list
    - compute.globalAddresses.list
    - compute.globalAddresses.setLabels
    - compute.instances.list
    - compute.instances.setLabels
    - compute.images.list
    - compute.images.setLabels
    - compute.machineTypes.get
    - compute.machineTypes.list
    - compute.networks.list
    - compute.regions.list
    - compute.snapshots.list
    - compute.snapshots.setLabels
    - compute.zones.list
    - iam.serviceAccounts.list
    - monitoring.timeSeries.list
    - storage.buckets.get
    - storage.buckets.getIamPolicy
    - storage.buckets.list
    - storage.buckets.update
Create service account. Official documentation - Service accounts | IAM Documentation | Google Cloud. Specify the role that you've created earlier and click Continue and then Done.
Grant access. For each project that needs to be added to the tenant, go to the IAM & Admin section in the Google Cloud Console, select IAM, and press the GRANT ACCESS button. Add the created service account and assign the created role to it.
Generate API key for your service account. Official documentation - Create a service account key | IAM Documentation | Google Cloud. Download it as a .json file. It is required on the stage of connection to Cavisson.