Google Cloud Storage Data Lake

Google Cloud Storage (GCS) is a popular web service for storing and accessing your data in the Google Cloud Platform infrastructure. It offers state-of-the-art performance and scalability, along with ensuring the security and privacy of your data.

RudderStack supports GCS data lake as a destination to which you can securely send your data.

User permissions for GCS data lake

To set up GCS data lake as a destination in RudderStack, you will need to create a new user role and grant the required permissions.

Creating the role and adding permissions

• Go to https://console.cloud.google.com/iam-admin/roles and click on CREATE ROLE.
• Then, fill in the details as shown:

• Fill the details and click on ADD PERMISSIONS.
• Under Filter permissions by role, select Storage Object Admin and grant the required permissions:

• The permission required to successfully use the GCS data lake destination is shown:

Copy
storage.objects.create

• Then, click on CREATE. This will successfully create the role.

Creating the service account and attaching the role

• Go to https://console.cloud.google.com/iam-admin/serviceaccounts.

• Then, select the project containing the dataset that you want to use.

• Next, click on CREATE SERVICE ACCOUNT.

• Fill in the details as shown below and click on CREATE.

• Then, select the previously created role and click on CONTINUE.

• Finally, after completing Steps 1 and 2, click on DONE.

Creating and downloading the JSON key

• Click on the three dots under Actions in the service account that you just created and select Manage keys, as shown:

• Click on ADD KEY, followed by Create new key, as shown:

• In the resulting pop-up, select JSON and click on CREATE.

• Finally, download this JSON file. This file is required while setting up the GCS data lake destination in RudderStack.

Setting up the GCS data lake destination

To start sending data to your GCS data lake, you will first need to add it as a destination in RudderStack and connect it to a data source. Once the destination is enabled, the events will automatically start flowing to your data lake.

Follow these steps to configure your GCS data lake as a destination in RudderStack:

• From your RudderStack dashboard, configure the data source. Then, select GCS Data Lake from the list of destinations.

• Assign a name to your destination and click on Next. You should then see the following screen:

Connection settings

Add the following credentials in the Connection Settings:

• Staging GCS Storage Bucket Name: Enter the name of the S3 bucket that will be used to store data before loading it into the S3 data lake.
• Prefix: If specified, RudderStack will create a folder in the bucket with this prefix and push all the data within that folder.
• Namespace: If specified, all the data for the destination will be pushed to https://storage.googleapis.com/<bucketName>/<prefix>/rudder-datalake/<namespace>.

• Credentials: Enter the content of the downloaded JSON key file in this field.

Finding your data in the GCS data lake

RudderStack converts your events into Parquet files and dumps them into the configured GCS bucket. Before dumping the events, RudderStack groups the files by the event name based on the time (UTC) they were received.

The folder structure is similar to the following format:

https://storage.googleapis.com/<bucketName>/<prefix>/rudder-datalake/<namespace>/<tableName>/YYYY/MM/DD/HH

As specified in the Connnection settings section above:

• <prefix> is the GCS prefix used while configuring the GCS data lake destination in RudderStack. If not specified, RudderStack will omit this value.
• <namespace> is the namespace specified in the destination settings. If not specified, RudderStack sets it to the source name.
• <tableName> is set to the event name.
• YYYY, MM, DD, and HH are replaced by the actual time values.

Use case

Suppose that RudderStack tracks the following two events:

RudderStack then converts these events into Parquet files and dumps them into the following folders:

Contact us

If you come across any issues while setting up using the GCS data lake destination, you can contact us or start a conversation on our Slack channel.