Cannot consume messages if they are not published using this library

[zRosenthal]

Describe the bug

Love the idea of this library as it drastically simplifies consuming from SQS. However in testing I am unable to get it to work unless the messages are published using this library.
This makes this library unusable in most enterprise environments where we consume events from a wide variety of sources many of which are not directly controlled by the team/app that is publishing those events.

Expected Behavior

Able to consume any json message

Current Behavior

Quick test routing events from AWS EventBridge

fail: AWS.Messaging.Serialization.EnvelopeSerializer[0]
MessageEnvelope instance is not valid
Id cannot be null or empty.
Source cannot be null.
Version cannot be null or empty.
MessageTypeIdentifier cannot be null or empty.
TimeStamp is not set.Message cannot be null.

Whats interesting about this is that the message from EventBridge actually does include Id, Source, Version however my understanding is that because that data is not contained in the detail node it is ignored.

Reproduction Steps

Publish a generic json object to an SQS queue without utilizing this library.

Possible Solution

Why are all of these fields required? Do we truly need them or can they be optional?

Can we allow custom mappers for the fields that are absolutely required? Could even ship with a prebuilt EventBridge mapper to utilize the existing properties in event.

Additional Information/Context

No response

AWS.Messaging (or related) package versions

AWS.Messaging 0.9.1

Targeted .NET Platform

.NET 8

Operating System and version

OSX Sonoma 14.1

[normj]

Thanks for the feedback @zRosenthal. This library is designed to by default expect messages to be in the CloudEvents specification. The publishers in this library publish in this specification. We also need someway to understand the type of message so we can map it to .NET type for serialization and mapping to handler. CloudEvents gave us placeholders for that information.

We did want to make the library extensible for non CloudEvents scenarios. You can inject your own implementation of the IEnvelopeSerializer and IMessageSerializer interfaces into the DI container. This allows you to define your own format of the messages. I would love to hear your feedback on adding your own serialization format to the library.

[AndrewTziAnChan]

At the innermost layer, the data attribute inside the CloudEvent JSON object contains a JSON serialization of the .NET object that was sent or received as the message.

For this framework, the data payload in the CloudEvent should just be a string (an escaped string at that), right? Is it a general AWS recommendation to serialize the data field this way when routing messages through the event bus? The CloudEvent spec seems to be open about the content type and even in an earlier blog post, the payload is a JSON object.

If some other microservice (maybe a lamdba or non-.Net app) tried to publish a message to be consumed up by a .NET service that utilizes this messaging framework, that message payload should ideally be stringified for smooth deserialization by the message framework. Would that be a misinterpretation of opinion and intent from this framework? A stronger opinion seems beneficial if the CloudEvent spec is going to be adopted because it wouldn't be ideal for applications that do not have access to this messaging framework to run into integration issues just because the format of the data happens to be slightly different.

e.g.

If the .NET application received this message (note the format of the data field):

{
  "version": "0",
  "id": "dbc1c73a-c51d-0c0e-ca61-ab9278974c58",
  "account": "1234567890",
  "time": "2023-05-23T12:38:46Z",
  "region": "us-east-1",
  "detail-type": "OrderPlaced",
  "source": "myapp.orders-service",
  "resources": [],
  "detail": {
    "specversion": "1.0",
    "id": "bba4379f-b764-4d90-9fb2-9f572b2b0b61",
    "source": "myapp.orders-service",
    "type": "OrderPlaced",
    "data": {
      "order_id": "c172a984-3ae5-43dc-8c3f-be080141845a",
      "customer_id": "dda98122-b511-4aaf-9465-77ca4a115ee6",
      "order_total": "120.00"
    },
    "time": "2024-01-01T17:31:00Z",
    "dataschema": "https://us-west-2.console.aws.amazon.com/events/home?region=us-west-2#/registries/discovered-schemas/schemas/myapp.orders-service%40OrderPlaced",
    "correlationid": "dddd9340-135a-c8c6-95c2-41fb8f492222",
    "domain": "ORDERS"
  }

Is the expectation that a non-serialized object (JSON object) in the payload is unexpected and needs to be explicitly handled?

[Tridus]

I ran into identical problems starting out with this. I've got a queue that receives events from EventBridge when a PutObject is completed in an S3 bucket. This library has no idea how to understand those messages, apparently because it expects the CloudEvents spec while EventBridge isn't publishing the event in that spec.

While I can inject new serializers in to get around that, it feels like its undermining a lot of the headline selling point of the library in the first place (that it makes working with SQS easy so you can focus on business logic, which is a great idea). Having a built in method to say "deserliaze this message into this class definition I've provided and decorated with [JsonPropertyName]" would alleviate the issue entirely.

That's something I definitely can do, but it feels like something a library like this should have a means to handle out of the box given that it's going to be a common usage scenario.

Thanks. :)

[Mohamed-Ahmed-Abdullah]

I needed a sample message to test my code, so I used this format:

{"id":"9fefe5d5-fbeb-4582-9645-e7d06a7b1e45","source":"/AmazonECS/Dev/193f9e06677d4ce1b7f6f7ebc78af48d","specversion":"1.0","type":"MessageModel.MyMessage","time":"2025-04-12T08:09:40.367408+00:00","data":"{\u0022Text\u0022:\u0022MessageText\u0022}"}

I also managed to create a Lambda function that publishes messages to the queue.

    const messageBody = JSON.stringify({
      id: uuidv4(),
      source: "/AmazonECS/Dev/193f9e06677d4ce1b7f6f7ebc78af48d", // Adjust if needed
      specversion: "1.0",
      type: "MessageModel.MyMessage",
      time: timestamp,
      data: JSON.stringify({
        Text: event.Text,
      }),
    });

    const params = {
      QueueUrl: queueUrl,
      MessageBody: messageBody,
    };

    const result = await sqs.sendMessage(params).promise();

However, I'm not sure what is the source field is and how to construct it.