Add SSL Databricks-Connect blog post.

Author: Thomas

_posts/2024-05-15-SSL-Issues-Databricks-Connect.md

@@ -1,86 +1,68 @@
 ---
 title: SSL Issues with Databricks Connect
-date: 2024-05-15 10:00:00 +0100
-categories: [Databricks]
-tags: [databricks, spark, databricks-connect, spark-connect, ssl, ]
+date: 2024-05-15 17:00:00 +0100
+categories: [Databricks, Databrick-Connect, Spark-Connect]
+tags: [databricks, spark, databricks-connect, spark-connect, ssl, gRPC, certificates, windows, python, pip, requests, certifi, pip-system-certs, python-certifi-win32]
 ---
 
-## The Issue
+## Issue
 
-If you are trying to access workspace files or files located in a repository folder (as described [here](https://learn.microsoft.com/en-us/azure/databricks/files/workspace-interact)) from a shared cluster in Databricks, you might run into the following error:
+If you are running Databricks-Connect locally on a Windows machine in a (company) network in which custom certificates (root & intermediary) are being used, you might run into the following (or a similar) error(s) when executing your code:
 
-`java.lang.SecurityException: User does not have permission SELECT on any file.`
-
-The underlying problem has to do with the [restrictions](https://docs.databricks.com/en/clusters/configure.html#shared-access-mode-limitations) you are facing when using shared clusters in Databricks.
+```text
+Handshake failed with fatal error SSL ERROR SSL: SSL routines :OPENSSL internal:CERTIFICATE VERIFY FAILED.
+```
 
-If you want to access Unity Catalog with a cluster the only two options regarding `Access Mode` currently are `Shared` or `Single User` though. If the latter is out of the question as it often happens in projects that i am participating in, then you will not be able to access workspace files from the only cluster option that is left.
+Root cause is that Databricks-Connect is using [gRPC](https://grpc.io/), which does (in its current implementation) not know about any custom/company certificates, even if they are correctly rolled out to your local Windows cert-store.
 
-## The Solution
+The problem is very similar to the SSL issues you run into when using pip/requests or any other module that is using [certifi](https://pypi.org/project/certifi/). On the requests/certifi/pip topic you can find numerous posts online, for example [this one](https://stackoverflow.com/questions/51390968/python-ssl-certificate-verify-error#:~:text=34-,Update,-python%2Dcertifi%2Dwin32). The solution(s) usually involve installing [pip-system-certs](https://pypi.org/project/pip-system-certs/) or the (meanwhile deprecated) [python-certifi-win32](https://pypi.org/project/python-certifi-win32/). These fixes however do not solve your gRPC issues in Databricks-Connect.
 
-One way of dealing with this problem is to use [Databricks Python SDK](https://databricks-sdk-py.readthedocs.io/en/latest/).
+## Solution
 
-### (1) Install the SDK on your Cluster
+### 1. Create a `.pem` file containing company-specific certificates
 
-First you need to install the SDK on your current cluster. There are many ways to install libraries on your cluster, the easiest one is to add this line of code in a notebook cell of its own:
+Assuming you have all the relevant company-specific certificates rolled out to your local Windows cert-store[^1] you can use this little python script to extract them:
 
 ```python
-%pip install databricks-sdk --upgrade
+import ssl
+
+context = ssl.create_default_context()
+der_certs = context.get_ca_certs(binary_form=True)
+pem_certs = [ssl.DER_cert_to_PEM_cert(der) for der in der_certs]
+
+with open('wincacerts.pem', 'w') as outfile:
+    for pem in pem_certs:
+        outfile.write(pem + '\n')
 ```
 
-### (2) Initialize, Connect & Authenticate
+This will create a file (wincacerts.pem) with all the certificates that are currently residing in your Windows cert-store.
 
-Then you need to connect to your workspace and authenticate. In my example i am assuming that you have a PAT stored in a secret scope called `keyvault` and the secret name is `databrickspat`. You can also provide a token in the code, but that is __never__ recommended.
+I suggest adding these certificates to the standard certificates that are shipped with [certifi](https://pypi.org/project/certifi)[^2]. To find those use:
 
-```python
-from databricks.sdk import WorkspaceClient
+```py
+import certifi
+
+print(certifi.where())
 
-w = WorkspaceClient(
-    host  = spark.conf.get("spark.databricks.workspaceUrl"),
-    token = dbutils.secrets.get(scope="keyvault", key="databrickspat")
-)
+>>> 'd:\repos\XXX\.venv\lib\site-packages\certifi\cacert.pem'
 ```
 
-### (3) Interact with Workspace Files
 
-```python
-# List DBFS with dbutils
-w.dbutils.fs.ls("/")
- 
-# ...or with 'dbfs'
-for file_ in w.dbfs.list("/"):
-    print(file_)
-    
-# List workspace files of a user
-for file_ in w.workspace.list("/Users/<username>", recursive=True):
-    print(file_.path)
-    
-# List repository files of a user
-for file_ in w.workspace.list("/Repos/<username>/<repo>"):
-    print(file_.path)
- 
-# Get contents of a yaml file stored in Repos/...    
-for line in w.workspace.download(path="/Repos/<username>/<repo>/.../catalog.yml"):
-    print(line.decode("UTF-8").replace("\n", ""))
-
-# Upload a (text) file to user repository folder
-import base64
-from databricks.sdk.service import workspace
-
-path = "/Repos/<username>/<repo>/.../test.yml"
-w.workspace.import_(content=base64.b64encode(("This is the file's content").encode()).decode(),
-    format=workspace.ImportFormat.AUTO,
-    overwrite=True,
-    path=path
-)
-```
 
-### Some Example Screenshots
+Open the file found at the returned location plus the file we created before in any text editor and append the contents of `cacert.pem` into `wincacerts.pem` via simple copy/paste. The order of certificates within this file doesn't matter. Afterwards save and move `wincacerts.pem` to any location in your Windows user home. For example: `C:\Users\<username>\certs\wincacerts.pem`
+
+### 2. Refer to custom certificates via environment variable(s)
 
-List files in DBFS:
-![List DBFS](/assets/img/dbfs.png)
+Now we just need to tell `gRPC` where our custom certificate file is located. The easiest approach is to use the environment variable `GRPC_DEFAULT_SSL_ROOTS_FILE_PATH`. You can set that environment variable either in VS-Code (for a specific project/terminal) or - the method i prefer - in your user's Windows environment variables[^3]:
+
+<img src="../assets/img/ssl_grpc_env.png" width="65%"/><br/>
+_Hint: After setting new environment variables you need to restart all open terminals/shells to make the changes effective._
+
+---
+:tada: **And that's about it!** :tada:
 
-List files in User/Workspace:
-![List User Files](/assets/img/user.png)
+Next time you run your Databricks-Connect code, the SSL handshake error should be gone.
 
-Show contents of a yaml file saved in user's repository folder:
-![Download File](/assets/img/yaml.png)
+[^1]: You can check your Windows cert-store via `Win+R -> certmgr.msc`.
+[^2]: In case you are missing [certifi](https://pypi.org/project/certifi) in your Python environment, run `pip install certifi` first.
+[^3]: You can set environment variables in Windows via `Win+R -> sysdm.cpl -> Advanced -> Environment Variables...`.