Elementary unstructured data validation tests run on top of Databricks AI Functions for Databricks users.
This guide provides details on the prerequisites to use Databricks AI Functions.
Databricks AI Functions are built-in SQL functions that allow you to apply AI capabilities directly to your data using SQL. These functions enable you to leverage large language models and other AI capabilities without complex setup or external dependencies, making them ideal for data validation tests.
Databricks AI functions can run on foundation models hosted in Databricks, external foundation models (like OpenAI’s models) and custom models.
Currently Elementary’s unstructured data validations support only foundation models hosted in Databricks. Adding support for external and custom models is coming soon.
Note: While developing the tests we worked with databricks-meta-llama-3-3-70b-instruct so we recommend using this model as a default when running unstructured data validation tests in Databricks.
When using AI functions, be aware that some models are limited to specific regions (US and EU). Make sure your Databricks workspace is in a supported region for the Databricks AI functions.