Skip to content

About

No description, website, or topics provided.

Resources

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

CONNECT Get Stream Updates From Databricks Notebook

Version: 1.0.0

The sample code in this folder demonstrates how to get stream updates with CONNECT data services (Cds) change broker and write those changes to a delta table in Databricks. In order to run this sample, you need to have a Databricks environment.

Developed against Python 3.10.12

Build Status

About the sample

This sample demonstrates how to use the change broker in CONNECT data services to get stream updates. In general, when making data available in Databricks from CONNECT data services, you should use the Virtual Tables feature. This is a no-code solution to getting stream and asset data into Databricks. However, using the change broker in addition to virtual tables can be useful if you need faster (less than 15 minute) updates on how a stream's data has changed.

This sample contains two notebooks:

  1. Get Updates from CONNECT Notebook.ipynb can run manually and walks you through the steps to get the updates. Use this notebook as a learning guide to understand the flow of getting updates.

  2. Get Updates from CONNECT Notebook - For Jobs.ipynb can be run via a databricks job. It cannot be run manually since it uses job parameters for certain connection information and settings. This sample doesn't walk through it's logic but is meant to show how you might implement the code in a job. Other than those differences, they use the same code and logic to accomplish getting stream updates.

Both samples assume that the chosen stream has a Value and Timestamp pair and that the values are numeric type. If your stream contains values of a type that cannot be converted to double, you will need to modify the code accordingly.

Getting started

  • Clone the GitHub repository or download the .ipnyb files
  • Import the notebook(s) into your Databricks environment. Follow the instructions at Import and export Databricks notebooks
  • For Get Updates from CONNECT Notebook.ipynb, follow the instructions in the notebook an run each code block in order.
  • For Get Updates from CONNECT Notebook - For Jobs.ipynb, create a scheduled Notebook Job in Databricks by following the instructions at Create and manage scheduled notebook jobs You will need to set the following as job parameters:
    • apiVersion - the API version for the signups and updates REST endpoints of CONNECT. (e.g. v1)
    • resource - the base CONNECT url for the region of your namespace. (e.g. https://uswe.datahub.connect.aveva.com)
    • tenantId - the tenant Id from CONNECT.
    • namespaceId - the namespace Id from CONNECT.
    • streamIds - the list of stream Ids to get updates for from CONNECT. This should be a comma separated list. (e.g. stream1, stream2)
    • catalog_name - the catalog name of the delta table you'd like to create and write updates to.
    • schema_name - the schema name of the delta table you'd like to create and write updates to.
    • table_name - the catalog name of the delta table you'd like to create and write updates to.

Creating the secret scope

This sample makes use of secret scopes in Databricks to securely store the credentials to CONNECT data services. To set up the secret scope, first create a client-credentials client in CONNECT data services. For this sample, the client-credentials client needs to be given a role that has read and write access to a CONNECT data services namespace. For instructions, see: Add a client-credentials client

You can then create the secret scope with one of two options:

  1. Use the Databricks CLI by following Create a secret in order to create two secrets within that secret scope called cdsclientid and cdsclientsecret containing the client id and secret generated from CONNECT data services

  2. Use the Databricks SDK for Python using the included sample code to create the secret scope and secrets.

Note: Option 2 requires temporarily entering your client credentials into the notebook as plain text. It's recommended to delete these credentials after the block is run.

Running the notebook

To run the Notebook(s), you first need to attach it to a compute resource (cluster or SQL warehouse). Once attached, you can run cells individually, run all cells, or schedule the notebook as a job.

Test the notebook

The last cell in the Get Updates from CONNECT Notebook.ipynb are for running tests so that you can test to make sure the whole notebook is working as expected. Because it tests the methods defined earlier in the notebook, you need to run the previous cells of the notebook before trying to run the tests. You will know the tests pass when the block succeeds without any exceptions.


For the main CONNECT data services samples page ReadMe
For the main AVEVA samples page ReadMe

About

No description, website, or topics provided.

Resources

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages