AWS Glue serde not available for non-name strategy

[Ronserruya]

I tried configuring the new aws glue serde to read messages encoded with the schema registry, but I can't see the glue serde in the value dropdown. Not sure if it's a bug or if I configured it wrong.

Set up
I tried following the example from https://github.com/provectus/kafkaui-glue-sr-serde/blob/main/docker-compose/setup-example.yaml

Folder structure:

 .
├──  vol
│  └──  kafkaui-glue-serde-1.0-SNAPSHOT-jar-with-dependencies.jar
└──  docker-compose.yml

---
version: '2'
services:

  kafka-ui:
    container_name: kafka-ui
    image: provectuslabs/kafka-ui:latest
    ports:
      - 8080:8080
    environment:
      AWS_ACCESS_KEY_ID: '****'
      AWS_SECRET_ACCESS_KEY: '***'
      AWS_SESSION_TOKEN: '***'

      kafka.clusters.0.name: Mycluster
      kafka.clusters.0.bootstrapServers: '***'

      kafka.clusters.0.serde.0.name: GlueSchemaRegistry
      kafka.clusters.0.serde.0.filePath: /glue-serde/kafkaui-glue-serde-1.0-SNAPSHOT-jar-with-dependencies.jar
      kafka.clusters.0.serde.0.className: com.provectus.kafka.ui.serdes.glue.GlueSerde
      kafka.clusters.0.serde.0.properties.region: us-east-1  #required
      kafka.clusters.0.serde.0.properties.registry: '***' #required, name of Glue Schema Registry

    volumes:
      - ./vol/:/glue-serde

The kafkaui-glue-serde-1.0.0.jar file from releases in https://github.com/provectus/kafkaui-glue-sr-serde didn't work at first, I guess it was missing the dependencies, so I built it myself (mvn package), to create the kafkaui-glue-serde-1.0-SNAPSHOT-jar-with-dependencies.jar file.

Screenshots

logs

ron@bash:~/Desktop/Projects/kafka_ui_test$ docker-compose up
[+] Running 1/1
 ⠿ Container kafka-ui  Recreated                                                                                                                        2.4s
Attaching to kafka-ui
kafka-ui  |  _   _ ___    __             _                _          _  __      __ _
kafka-ui  | | | | |_ _|  / _|___ _ _    /_\  _ __ __ _ __| |_  ___  | |/ /__ _ / _| |_____
kafka-ui  | | |_| || |  |  _/ _ | '_|  / _ \| '_ / _` / _| ' \/ -_) | ' </ _` |  _| / / _`|
kafka-ui  |  \___/|___| |_| \___|_|   /_/ \_| .__\__,_\__|_||_\___| |_|\_\__,_|_| |_\_\__,|
kafka-ui  |                                  |_|
kafka-ui  |
kafka-ui  | 2023-01-12 15:38:36,912 INFO  [background-preinit] o.h.v.i.u.Version: HV000001: Hibernate Validator 6.2.5.Final
kafka-ui  | 2023-01-12 15:38:36,951 INFO  [main] c.p.k.u.KafkaUiApplication: Starting KafkaUiApplication using Java 17.0.5 on d01713c83787 with PID 1 (/kafka-ui-api.jar started by kafkaui in /)
kafka-ui  | 2023-01-12 15:38:36,952 DEBUG [main] c.p.k.u.KafkaUiApplication: Running with Spring Boot v2.7.5, Spring v5.3.23
kafka-ui  | 2023-01-12 15:38:36,953 INFO  [main] c.p.k.u.KafkaUiApplication: No active profile set, falling back to 1 default profile: "default"
kafka-ui  | 2023-01-12 15:38:41,813 DEBUG [main] c.p.k.u.s.SerdesInitializer: Configuring serdes for cluster Mycluster
kafka-ui  | 2023-01-12 15:38:41,833 INFO  [main] c.p.k.u.s.SerdesInitializer: Loading custom serde GlueSchemaRegistry
kafka-ui  | 2023-01-12 15:38:43,417 INFO  [main] c.a.s.s.c.c.GlueSchemaRegistryConfiguration: endpoint key is not present in the configs
kafka-ui  | 2023-01-12 15:38:43,417 INFO  [main] c.a.s.s.c.c.GlueSchemaRegistryConfiguration: registry.name key is not present in the configs
kafka-ui  | 2023-01-12 15:38:43,417 INFO  [main] c.a.s.s.c.c.GlueSchemaRegistryConfiguration: description key is not present in the configs
kafka-ui  | 2023-01-12 15:38:43,417 INFO  [main] c.a.s.s.c.c.GlueSchemaRegistryConfiguration: avroRecordType key is not present in the configs
kafka-ui  | 2023-01-12 15:38:43,417 INFO  [main] c.a.s.s.c.c.GlueSchemaRegistryConfiguration: protobufMessageType key is not present in the configs
kafka-ui  | 2023-01-12 15:38:43,417 INFO  [main] c.a.s.s.c.c.GlueSchemaRegistryConfiguration: compatibility key is not present in the configs
kafka-ui  | 2023-01-12 15:38:43,419 INFO  [main] c.a.s.s.c.c.GlueSchemaRegistryConfiguration: compression key is not present in the configs
kafka-ui  | 2023-01-12 15:38:43,419 INFO  [main] c.a.s.s.c.c.GlueSchemaRegistryConfiguration: schemaAutoRegistrationEnabled key is not present in the configs
kafka-ui  | 2023-01-12 15:38:43,419 INFO  [main] c.a.s.s.c.c.GlueSchemaRegistryConfiguration: schemaAutoRegistrationEnabled is not defined in the properties. Using the default value false
kafka-ui  | 2023-01-12 15:38:43,419 INFO  [main] c.a.s.s.c.c.GlueSchemaRegistryConfiguration: jacksonSerializationFeatures key is not present in the configs
kafka-ui  | 2023-01-12 15:38:43,419 INFO  [main] c.a.s.s.c.c.GlueSchemaRegistryConfiguration: jacksonDeserializationFeatures key is not present in the configs
kafka-ui  | 2023-01-12 15:38:43,419 INFO  [main] c.a.s.s.c.c.GlueSchemaRegistryConfiguration: tags key is not present in the configs
kafka-ui  | 2023-01-12 15:38:43,420 INFO  [main] c.a.s.s.c.c.GlueSchemaRegistryConfiguration: Tags value is not defined in the properties. No tags are assigned
kafka-ui  | 2023-01-12 15:38:43,420 INFO  [main] c.a.s.s.c.c.GlueSchemaRegistryConfiguration: metadata key is not present in the configs
kafka-ui  | 2023-01-12 15:38:43,420 INFO  [main] c.a.s.s.c.c.GlueSchemaRegistryConfiguration: userAgentApp key is not present in the configs
kafka-ui  | 2023-01-12 15:38:43,420 INFO  [main] c.a.s.s.c.c.GlueSchemaRegistryConfiguration: secondaryDeserializer key is not present in the configs
kafka-ui  | 2023-01-12 15:38:43,420 INFO  [main] c.a.s.s.c.c.GlueSchemaRegistryConfiguration: cacheSize key is not present in the configs
kafka-ui  | 2023-01-12 15:38:43,420 INFO  [main] c.a.s.s.c.c.GlueSchemaRegistryConfiguration: Cache Size is not found, using default 200
kafka-ui  | 2023-01-12 15:38:43,420 INFO  [main] c.a.s.s.c.c.GlueSchemaRegistryConfiguration: timeToLiveMillis key is not present in the configs
kafka-ui  | 2023-01-12 15:38:43,420 INFO  [main] c.a.s.s.c.c.GlueSchemaRegistryConfiguration: Cache Time to live is not found, using default 86400000
kafka-ui  | 2023-01-12 15:38:44,806 INFO  [main] o.s.b.a.e.w.EndpointLinksResolver: Exposing 2 endpoint(s) beneath base path '/actuator'
kafka-ui  | 2023-01-12 15:38:45,147 INFO  [main] o.s.b.a.s.r.ReactiveUserDetailsServiceAutoConfiguration:
kafka-ui  |
kafka-ui  | Using generated security password: ****
kafka-ui  |
kafka-ui  | 2023-01-12 15:38:45,444 WARN  [main] c.p.k.u.c.a.DisabledAuthSecurityConfig: Authentication is disabled. Access will be unrestricted.
kafka-ui  | 2023-01-12 15:38:46,437 INFO  [main] o.s.b.w.e.n.NettyWebServer: Netty started on port 8080
kafka-ui  | 2023-01-12 15:38:46,494 INFO  [main] c.p.k.u.KafkaUiApplication: Started KafkaUiApplication in 10.636 seconds (JVM running for 11.868)
kafka-ui  | 2023-01-12 15:38:46,564 DEBUG [parallel-1] c.p.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: Mycluster
kafka-ui  | 2023-01-12 15:38:46,608 INFO  [parallel-1] o.a.k.c.a.AdminClientConfig: AdminClientConfig values:
kafka-ui  | 	bootstrap.servers = [****]
kafka-ui  | 	client.dns.lookup = use_all_dns_ips
kafka-ui  | 	client.id = kafka-ui-admin-client-1673537926591
kafka-ui  | 	connections.max.idle.ms = 300000
kafka-ui  | 	default.api.timeout.ms = 60000
kafka-ui  | 	metadata.max.age.ms = 300000
kafka-ui  | 	metric.reporters = []
kafka-ui  | 	metrics.num.samples = 2
kafka-ui  | 	metrics.recording.level = INFO
kafka-ui  | 	metrics.sample.window.ms = 30000
kafka-ui  | 	receive.buffer.bytes = 65536
kafka-ui  | 	reconnect.backoff.max.ms = 1000
kafka-ui  | 	reconnect.backoff.ms = 50
kafka-ui  | 	request.timeout.ms = 30000
kafka-ui  | 	retries = 2147483647
kafka-ui  | 	retry.backoff.ms = 100
kafka-ui  | 	sasl.client.callback.handler.class = null
kafka-ui  | 	sasl.jaas.config = null
kafka-ui  | 	sasl.kerberos.kinit.cmd = /usr/bin/kinit
kafka-ui  | 	sasl.kerberos.min.time.before.relogin = 60000
kafka-ui  | 	sasl.kerberos.service.name = null
kafka-ui  | 	sasl.kerberos.ticket.renew.jitter = 0.05
kafka-ui  | 	sasl.kerberos.ticket.renew.window.factor = 0.8
kafka-ui  | 	sasl.login.callback.handler.class = null
kafka-ui  | 	sasl.login.class = null
kafka-ui  | 	sasl.login.connect.timeout.ms = null
kafka-ui  | 	sasl.login.read.timeout.ms = null
kafka-ui  | 	sasl.login.refresh.buffer.seconds = 300
kafka-ui  | 	sasl.login.refresh.min.period.seconds = 60
kafka-ui  | 	sasl.login.refresh.window.factor = 0.8
kafka-ui  | 	sasl.login.refresh.window.jitter = 0.05
kafka-ui  | 	sasl.login.retry.backoff.max.ms = 10000
kafka-ui  | 	sasl.login.retry.backoff.ms = 100
kafka-ui  | 	sasl.mechanism = GSSAPI
kafka-ui  | 	sasl.oauthbearer.clock.skew.seconds = 30
kafka-ui  | 	sasl.oauthbearer.expected.audience = null
kafka-ui  | 	sasl.oauthbearer.expected.issuer = null
kafka-ui  | 	sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
kafka-ui  | 	sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
kafka-ui  | 	sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
kafka-ui  | 	sasl.oauthbearer.jwks.endpoint.url = null
kafka-ui  | 	sasl.oauthbearer.scope.claim.name = scope
kafka-ui  | 	sasl.oauthbearer.sub.claim.name = sub
kafka-ui  | 	sasl.oauthbearer.token.endpoint.url = null
kafka-ui  | 	security.protocol = PLAINTEXT
kafka-ui  | 	security.providers = null
kafka-ui  | 	send.buffer.bytes = 131072
kafka-ui  | 	socket.connection.setup.timeout.max.ms = 30000
kafka-ui  | 	socket.connection.setup.timeout.ms = 10000
kafka-ui  | 	ssl.cipher.suites = null
kafka-ui  | 	ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
kafka-ui  | 	ssl.endpoint.identification.algorithm = https
kafka-ui  | 	ssl.engine.factory.class = null
kafka-ui  | 	ssl.key.password = null
kafka-ui  | 	ssl.keymanager.algorithm = SunX509
kafka-ui  | 	ssl.keystore.certificate.chain = null
kafka-ui  | 	ssl.keystore.key = null
kafka-ui  | 	ssl.keystore.location = null
kafka-ui  | 	ssl.keystore.password = null
kafka-ui  | 	ssl.keystore.type = JKS
kafka-ui  | 	ssl.protocol = TLSv1.3
kafka-ui  | 	ssl.provider = null
kafka-ui  | 	ssl.secure.random.implementation = null
kafka-ui  | 	ssl.trustmanager.algorithm = PKIX
kafka-ui  | 	ssl.truststore.certificates = null
kafka-ui  | 	ssl.truststore.location = null
kafka-ui  | 	ssl.truststore.password = null
kafka-ui  | 	ssl.truststore.type = JKS
kafka-ui  |
kafka-ui  | 2023-01-12 15:38:46,885 INFO  [parallel-1] o.a.k.c.u.AppInfoParser: Kafka version: 3.3.1
kafka-ui  | 2023-01-12 15:38:46,885 INFO  [parallel-1] o.a.k.c.u.AppInfoParser: Kafka commitId: e23c59d00e687ff5
kafka-ui  | 2023-01-12 15:38:46,885 INFO  [parallel-1] o.a.k.c.u.AppInfoParser: Kafka startTimeMs: 1673537926879
kafka-ui  | 2023-01-12 15:38:50,739 DEBUG [parallel-6] c.p.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: Mycluster

[github-actions]

Hello there Ronserruya! 👋

Thank you and congratulations 🎉 for opening your very first issue in this project! 💖

In case you want to claim this issue, please comment down below! We will try to get back to you as soon as we can. 👀

[Haarolean]

Hey, thanks for reaching out.
We'll take a look soon.

[Haarolean]

@Ronserruya does the schema name in glue represent the topic name exactly?
The default glue serde behavior is the following: Upon deserializing the message, kafka-ui will try to look up a schema with the name "%topicname%" (that's a default one for GlueSchemaRegistryKafkaSerializer), so in case it doesn't exist -- it assumes that glue serde is not available for the topic.
On a side note, you're able to alter the pattern used for schema lookup by adjusting the following env vars:
kafka.clusters.0.serde.0.properties.keySchemaNameTemplate and kafka.clusters.0.serde.0.properties.valueSchemaNameTemplate.

[Ronserruya]

Hey @Haarolean , thank you for the quick response. I wasn't aware that that was a requirement. I created a test topic and schema with matching names and validated that it does work.

However there are 2 issues:

1. The resulting decoded data injects the namespaces into the data. I decoded similar avro msgs using python, kafka-connect, and spark and didn't encounter this behavior.
   I sent this msg: {"name": {"raw": null, "first": "ron", "full": "ron serruya", "last": "serruya"}, "age": 3}
   Encoded with this schema:

{
  "type": "record",
  "name": "generation",
  "namespace": "top_level",
  "fields": [
    {
      "name": "name",
      "type": [
        {
          "type": "record",
          "name": "name",
          "namespace": "top_level.generation",
          "fields": [
            {
              "name": "raw",
              "type": [
                "string",
                "null"
              ]
            },
            {
              "name": "first",
              "type": "string"
            },
            {
              "name": "full",
              "type": "string"
            },
            {
              "name": "last",
              "type": "string"
            }
          ]
        },
        "null"
      ]
    },
    {
      "name": "age",
      "type": "int"
    }
  ]
}

Kafka-ui showed this decoded msg

{
	"name": {
		"top_level.generation.name": {
			"raw": null,
			"first": "ron",
			"full": "ron serruya",
			"last": "serruya"
		}
	},
	"age": 3
}

(let me know if you want to open a different issue for this, since its not exactly related)
2. This method of fetching the schema using the topic name is not really usable for us since we have many topics and schemas that don't match 1to1 (and can't be simply formatted like %s-value-schema), also, how does that work with multiple schemas per topic?

When a message is encoded using the aws glue schema registry format, the schema UUID is inserted from from bytes 2-17, so decoders like the python one or the one used in kafka connect just get the schema UUID from those bytes and fetch the schema from aws by ID, without needing the topic name.
See python, aws java library

Looking at the code in your deserialize method I don't see topic being used, so I suspect that by using the GlueSchemaRegistryDeserializationFacade class your code already does this upon desiralization, so all that needs to be done is to "unblock" the need for the topic name to match a schema.

Forgive me if I read the code incorrectly, being a python dev I lost myself in the sea of facades and interfaces 😅