Skip to content

Python: Help running [realtime] call automation python sample #11237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
moonbox3 opened this issue Mar 27, 2025 · 5 comments
Closed

Python: Help running [realtime] call automation python sample #11237

moonbox3 opened this issue Mar 27, 2025 · 5 comments
Assignees
@moonbox3
Labels
ai connector Anything related to AI connectors python Pull requests for the Python Semantic Kernel

Comments

@moonbox3
Copy link
Contributor

moonbox3 commented Mar 27, 2025 •
edited
Loading

Discussed in #11092

Originally posted by oradcliffe March 21, 2025
Has anyone successfully run the call automation sample? I am getting an error "DefaultAzureCredential failed to retrieve a token from the included credentials. Attempted credentials: EnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured."

As far as I can tell, my environmental variables are configured. Everything else seems to work, dev tunnel is up, app runs, event grid hands call off to the (local) app, but have an issue connecting to openAI realtime model.

If there are any gotchas that aren't called out in the docs I might be missing, I would love to hear about it.

Hi @moonbox3 - I am trying to run the call automation sample, found here. I'm using Azure Open AI with a realtime model deployed.

Here's what I am seeing most recently - previously I was getting a message that I didn't have environmental credentials but after throwing everything away and starting fresh, I am seeing this:

[2025-03-27 16:36:55,387] INFO in call_automation: incoming call handler caller id: +14061234567
[2025-03-27 16:36:55,387] INFO in call_automation: callback url: https://XXXXXX-8080.use.devtunnels.ms/api/callbacks/2fe6b126-2717-41b3-91ac-1588695a66be?callerId=%2B14061234567
[2025-03-27 16:36:55,387] INFO in call_automation: websocket url: wss://XXXXXX-8080.use.devtunnels.ms/ws
[2025-03-27 16:36:57,438] INFO in call_automation: Answered call for connection id: 21005880-e012-4fb9-85c5-7915e7319bfb
[2025-03-27 16:36:57 +0000] [8004] [INFO] 127.0.0.1:45576 POST /api/incomingCall 1.1 200 - 2056148
[2025-03-27 16:36:58,836] INFO in call_automation: Client connected to WebSocket
[2025-03-27 16:37:00,433] ERROR in app: Exception on websocket /ws
Traceback (most recent call last):
File "/home/codespace/.cache/uv/environments-v2/call-automation-535d4a790269f4c3/lib/python3.12/site-packages/quart/app.py", line 1476, in handle_websocket
return await self.full_dispatch_websocket(websocket_context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/codespace/.cache/uv/environments-v2/call-automation-535d4a790269f4c3/lib/python3.12/site-packages/quart/app.py", line 1525, in full_dispatch_websocket
result = await self.handle_user_exception(error)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/codespace/.cache/uv/environments-v2/call-automation-535d4a790269f4c3/lib/python3.12/site-packages/quart/app.py", line 1059, in handle_user_exception
raise error
File "/home/codespace/.cache/uv/environments-v2/call-automation-535d4a790269f4c3/lib/python3.12/site-packages/quart/app.py", line 1523, in full_dispatch_websocket
result = await self.dispatch_websocket(websocket_context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/codespace/.cache/uv/environments-v2/call-automation-535d4a790269f4c3/lib/python3.12/site-packages/quart/app.py", line 1613, in dispatch_websocket
return await self.ensure_async(handler)(**websocket_.view_args) # type: ignore[return-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspaces/semantic-kernel/python/samples/demos/call_automation/call_automation.py", line 181, in ws
async with client(settings=settings, create_response=True, kernel=kernel):
File "/home/codespace/.cache/uv/environments-v2/call-automation-535d4a790269f4c3/lib/python3.12/site-packages/semantic_kernel/connectors/ai/realtime_client_base.py", line 127, in aenter
await self.create_session(self._chat_history, self._settings)
File "/home/codespace/.cache/uv/environments-v2/call-automation-535d4a790269f4c3/lib/python3.12/site-packages/semantic_kernel/connectors/ai/open_ai/services/_open_ai_realtime.py", line 889, in create_session
self.connection = await self.client.beta.realtime.connect(model=self.ai_model_id).enter()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/codespace/.cache/uv/environments-v2/call-automation-535d4a790269f4c3/lib/python3.12/site-packages/openai/resources/beta/realtime/realtime.py", line 379, in aenter
await connect(
File "/home/codespace/.cache/uv/environments-v2/call-automation-535d4a790269f4c3/lib/python3.12/site-packages/websockets/asyncio/client.py", line 446, in await_impl
await self.connection.handshake(*self.handshake_args)
File "/home/codespace/.cache/uv/environments-v2/call-automation-535d4a790269f4c3/lib/python3.12/site-packages/websockets/asyncio/client.py", line 104, in handshake
raise self.protocol.handshake_exc
File "/home/codespace/.cache/uv/environments-v2/call-automation-535d4a790269f4c3/lib/python3.12/site-packages/websockets/client.py", line 340, in parse
self.process_response(response)
File "/home/codespace/.cache/uv/environments-v2/call-automation-535d4a790269f4c3/lib/python3.12/site-packages/websockets/client.py", line 151, in process_response
raise InvalidStatus(response)
websockets.exceptions.InvalidStatus: server rejected WebSocket connection: HTTP 404
[2025-03-27 16:37:00 +0000] [8004] [INFO] 127.0.0.1:45576 GET /ws 1.1 500 265 1608268
[2025-03-27 16:37:03,018] INFO in call_automation: Received Event:-> Microsoft.Communication.ParticipantsUpdated, Correlation Id:-> 1b07f579-df00-470e-ac73-4281ed03f4ef, CallConnectionId:-> 21005880-e012-4fb9-85c5-7915e7319bfb
[2025-03-27 16:37:03 +0000] [8004] [INFO] 127.0.0.1:34124 POST /api/callbacks/2fe6b126-2717-41b3-91ac-1588695a66be 1.1 200 - 1652
[2025-03-27 16:37:04,196] INFO in call_automation: Received Event:-> Microsoft.Communication.CallConnected, Correlation Id:-> 1b07f579-df00-470e-ac73-4281ed03f4ef, CallConnectionId:-> 21005880-e012-4fb9-85c5-7915e7319bfb
[2025-03-27 16:37:04,283] INFO in call_automation: MediaStreamingSubscription:--> {'additional_properties': {}, 'id': '5cfaaf6b-4675-4f82-8502-221c50cd4711', 'state': 'inactive', 'subscribed_content_types': ['audio']}
[2025-03-27 16:37:04,283] INFO in call_automation: Received CallConnected event for connection id: 21005880-e012-4fb9-85c5-7915e7319bfb
[2025-03-27 16:37:04,283] INFO in call_automation: CORRELATION ID:--> 1b07f579-df00-470e-ac73-4281ed03f4ef
[2025-03-27 16:37:04,283] INFO in call_automation: CALL CONNECTION ID:--> 21005880-e012-4fb9-85c5-7915e7319bfb
[2025-03-27 16:37:04 +0000] [8004] [INFO] 127.0.0.1:34140 POST /api/callbacks/2fe6b126-2717-41b3-91ac-1588695a66be 1.1 200 - 88002
[2025-03-27 16:37:04,655] INFO in call_automation: Received Event:-> Microsoft.Communication.MediaStreamingFailed, Correlation Id:-> 1b07f579-df00-470e-ac73-4281ed03f4ef, CallConnectionId:-> 21005880-e012-4fb9-85c5-7915e7319bfb
[2025-03-27 16:37:04,655] INFO in call_automation: Code:->400, Subcode:-> 8581
[2025-03-27 16:37:04,656] INFO in call_automation: Message:->Action failed, Transport url is not valid or web socket server is not operational.

For what it's worth, I've had to do a lot of trial and error to get this far. Here are my notes from a second go:

Re-cloned repo
Opened in dev container
Install python, python debugger extensions
Open Bash
Install UV
- curl -LsSf https://astral.sh/uv/install.sh | sh

/home/codespace/bin/devtunnel user login
/home/codespace/bin/devtunnel create --allow-anonymous
/home/codespace/bin/devtunnel port create -p 8080
/home/codespace/bin/devtunnel host

Take note of 'connect via browser'
e.g. https://XXXXXX-8080.use.devtunnels.ms

Open second terminal (leave dev tunnel running)
cd python/samples/demos/call_automation/

Rename .env.example to .env
- Fill in variables
- Need a realtime model
- API version '2025-03-01-preview'

uv run --env-file .env call_automation.py

Update event grid with tunnel URL (/api/incomingCall)
e.g. https://XXXXXX-8080.use.devtunnels.ms/api/incomingCall
@moonbox3 moonbox3 added ai connector Anything related to AI connectors python Pull requests for the Python Semantic Kernel labels Mar 27, 2025
@moonbox3
Copy link
Contributor Author

Hi @oradcliffe,

We've got a few people who are running into this same issue. I spent some time and have been working through the resource setup, and I am also hitting that issue, too. I will continue to look into it, at least it's reproducible.

@moonbox3
Copy link
Contributor Author

Hi @oradcliffe,

Alright, I got it working. Your current steps and config all look to be right. I believe the issue you're running into is that the API VERSION 2025-03-01-preview does not work for Azure OpenAI realtime deployments. I was also using that version and once I set it back to 2024-10-01-preview, along with adjusting my AOAI resource deployment, allowed the sample to work properly.

@moonbox3 moonbox3 assigned moonbox3 and unassigned eavanvalkenburg Mar 28, 2025
@moonbox3 moonbox3 removed the triage label Mar 28, 2025
@moonbox3
Copy link
Contributor Author

Out of this, I've also opened a small PR with some more detailed steps for others going through these same motions - these added steps would have helped set up the demo faster. #11249

@oradcliffe
Copy link

@moonbox3 I'll give this a shot today! Thank you so much. Next steps will be to move this into an Azure Container App for a more holistic Azure demo, and adding some extra data sources like sample product info. It's a very cool use case.

@oradcliffe
Copy link

@moonbox3 that worked! Only needed to change the API version. Thanks a ton for catching that.

@moonbox3 moonbox3 closed this as completed Apr 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@moonbox3 moonbox3

Labels
ai connector Anything related to AI connectors python Pull requests for the Python Semantic Kernel
Projects
Semantic Kernel
Archived in project
Milestone
No milestone
Development

No branches or pull requests
4 participants
@oradcliffe @eavanvalkenburg @moonbox3 @markwallace-microsoft