Serverless framework support

[datar-ai]

When using aws-xray-sdk 0.92 on Lambda to trace a Flask application, adding the Flask middleware results in this log and traceback.

[1509032914734] Traceback (most recent call last):
[1509032914734] File "/var/task/flask/app.py", line 1982, in wsgi_app
[1509032914734] response = self.full_dispatch_request()
[1509032914734] File "/var/task/flask/app.py", line 1615, in full_dispatch_request
[1509032914734] return self.finalize_request(rv)
[1509032914734] File "/var/task/flask/app.py", line 1632, in finalize_request
[1509032914734] response = self.process_response(response)
[1509032914734] File "/var/task/flask/app.py", line 1856, in process_response
[1509032914734] response = handler(response)
[1509032914734] File "/var/task/aws_xray_sdk/ext/flask/middleware.py", line 59, in _after_request
[1509032914734] segment.put_http_meta(http.STATUS, response.status_code)
[1509032914734] File "/var/task/aws_xray_sdk/core/models/facade_segment.py", line 42, in put_http_meta
[1509032914734] raise FacadeSegmentMutationException(MUTATION_UNSUPPORTED_MESSAGE)
[1509032914734] aws_xray_sdk.core.exceptions.exceptions.FacadeSegmentMutationException: FacadeSegments cannot be mutated.
[1509032914734] FacadeSegments cannot be mutated.

Please check the forums for the detail info :
https://forums.aws.amazon.com/thread.jspa?messageID=803898&tstart=0

[haotianw465]

Hi, which serverless framework are you using?

[datar-ai]

I use zappa , and python flask framework.

[haotianw465]

The long term solution is to have the middleware directly add all http metadata to the root segment generated by Lambda worker. The SDK currently doesn't have access to this segment right now and we don't have an ETA yet.

The workaround is to have middleware to only generate subsegments when running on Lambda. This causes a problem in the service graph. The color of the Lambda function node in the service graph is decided by status code in lambda segments. And this status code only indicates if the lambda function runs or not. It doesn't respect the actual status code returned by your web application running within that lambda function. If you have correct IAM role setup and no syntax error on your application code, most likely you will always see an all green service graph node despite a lot of 4xx returned by your web app.

This workaround makes it difficult to discover issues and we cannot call this workaround "support for serverless". But we are open to here any suggestions. If such workaround is acceptable we can make it an opt-in so that at least FacadeSegmentMutationException is avoided.

[Sniedes722]

patching the libraries seemed to solve this issue for me, also using lambda, flask, & zappa.

from aws_xray_sdk.core import xray_recorder, patch_all
from aws_xray_sdk.core.context import Context
from aws_xray_sdk.ext.flask.middleware import XRayMiddleware

patch_all()

and then this was how i wrapped it inside the flask application:

xray_recorder.configure(service='Your Service Name', sampling=False, context=Context())
XRayMiddleware(app, xray_recorder)

[Sniedes722]

also enabled in zappa config checkout: https://github.com/Miserlou/Zappa#aws-x-ray

[iandees]

For others running into this issue, @Sniedes722's combination of patching and enabling x-ray in Zappa worked for me. I'm not sure why, but it didn't work until I added the sampling=False, context=Context() stuff.

[Sniedes722]

@iandees I can also confirm that @haotianw465 is completely correct about getting green OK status codes in x-ray when the API is returning 400 statuses with this workaround

[xplorld]

@haotianw465 What are the consequences if we use our own Context() instead of using SDK-provided Lambda context?

[haotianw465]

@xplorld the recorder class intentionally exposes an API to allow users to configure their own context management https://github.com/aws/aws-xray-sdk-python/blob/master/aws_xray_sdk/core/recorder.py#L75.

You can have your own implementation of the context() used in Lambda container which creates another type of facadesegment which can be modified freely and replicate everything it has to a child subsegment.

Please let me know if you need any help on implementing a customized context management that specifically fits your use case.

[haotianw465]

Hi all, API Gateway with active X-Ray integration is launched. You can see more details here: https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-xray.html.

Feel free to leave any feedback on the new customer experience and we are happy to discuss a workaround based on those new feedback. We will also be working with Lambda team based on those new feedback to better improve the overall experience. Thank you for your patience.

I see the a dangling subsegment even when calling configure before patching third party libraries.

from`aws_xray_sdk.core import xray_recorder, patch_all
from aws_xray_sdk.core.context import Context
from aws_xray_sdk.ext.flask.middleware import XRayMiddleware

app = connexion.App(__name__, specification_dir='./swagger/')
app.app.json_encoder = encoder.JSONEncoder
app.add_api('swagger.yaml', arguments={'title': 'Swagger Petstore'})

xray_recorder.configure(service=' python-flask-se-development', sampling=False, context=Context())
XRayMiddleware(app.app, xray_recorder)

patch_all()