Sttp model does not describe how to match different responses to different models

[chameleon82]

Usually in REST applications one endpoint provide multiple models for response, generally at least one model for successful response and one for error.
It could be very useful to able to describe every expected answer and match it to client model.

One of the cases to use is able to create autogenerated clients. Here is example how model describe answers in OpenApi
https://github.com/OpenAPITools/openapi-generator/blob/master/samples/client/petstore/scala-akka/src/main/scala/org/openapitools/client/api/PetApi.scala#L38

[adamw]

@chameleon82 This is currently done in sttp-client:
https://sttp.readthedocs.io/en/latest/responses/body.html. In sttp-client, the request contains a specification of what to do with the response; this usually is an Either[X, Y], where the left describes the error case, and the right the success case.

For generating servers and documentation, this is also covered by sttp-tapir: https://tapir-scala.readthedocs.io/en/latest/endpoint/basics.html. There, the endpoint description contains information on how to map responses to errors and success values.

Would you somehow see a new abstraction in sttp-model? What kind of use-cases would it cover?

[chameleon82]

@adamw my concern about request model is to be more self-described. sttp-client cover only success/unsuccess case, but, for example, i can specify endpoint with the next expected mappings:

POST /create/something

200 -> SomethingModel
400 -> TechnicalProblemsModel, e.g. field "name" is missed
409 -> ValidationErrorsModel, e.g. Something with "name" already exists
422 -> ValidationErrorsModel

Of course, i can parse it to necessary models with additional callback on Left. But would it better if I can describe it with a model?

[adamw]

I think this is doable right now with sttp-client. For example, you can create the following response specification:

sealed trait MyModel
  case class SomethingModel() extends MyModel
  case class TechnicalProblemsModel() extends MyModel
  case class ValidationErrorsModel() extends MyModel
  case class ErrorModel() extends MyModel

  val asMyJson: ResponseAs[Either[ResponseError[circe.Error], MyModel], Nothing] = fromMetadata { meta =>
    meta.code match {
      case StatusCode.Ok         => asJson[SomethingModel]
      case StatusCode.BadRequest => asJson[TechnicalProblemsModel]
      case StatusCode.Conflict   => asJson[ValidationErrorsModel]
      case _                     => asJson[ErrorModel]
    }
  }

is this something as you had in mind?

[chameleon82]

@adamw Yes, exact that

[adamw]

@chameleon82 great, so - is what is currently available in sttp-client enough for your needs, or do you have a use-case that's not covered?

Maybe you have a good idea on how some of this could be moved to sttp-model :)

[chameleon82]

@adamw I tried next example

libraryDependencies ++= Seq(
  "com.softwaremill.sttp.client" %% "core" % "2.0.0-RC10",
  "com.softwaremill.sttp.client" %% "circe" % "2.0.0-RC10",
  "io.circe" %% "circe-generic" % "0.12.3"
)

import sttp.client._
import sttp.model.StatusCode
import sttp.client.circe._
import io.circe.generic.auto._

sealed trait MyModel
case class ItemModel(id: Int, node_id: String, name: String, full_name: String) extends MyModel
case class ResultModel(total_count: Int, incomplete_results: Boolean, items: List[ItemModel]) extends MyModel
case class ErrorModel(message: String) extends MyModel

object MyApp extends App {

  // Model Defintion
  def getRepositories(query: String, sort: Option[String] = None)
  :  RequestT[Identity,Either[ResponseError[io.circe.Error], MyModel], Nothing] = {
    basicRequest
      .get(uri"https://api.github.com/search/repositories?q=$query&sort=$sort")
      .response {
        fromMetadata { meta =>
          meta.code match {
            case StatusCode.Ok => asJson[ResultModel]
            case StatusCode.BadRequest => asJson[ErrorModel]
          }
        }
      }
  }

  implicit val backend = HttpURLConnectionBackend()
  val response = getRepositories(query = "http language:scala").send()

  response.body match {
    case Left(ex) => throw ex
    case Right(model) => println(model)
  }

}

It wont compile due expecting MyModel rather than ResultModel/ErrorModel. May be i did something wrong here.

But what I see here is that asJson method (and similar to it methods) can be extracted with some base trait SttpSerializerApi into the model. Right now i have to specify serializer implementation first, but actually i want to define model free from actual client and serializer implementation and define it separately.

For example above it could be transformed into something like this:

class RepositoryApi(implicit serializer: sttp.model.SerializerApi) {
  def getRepositories(query: String, sort: Option[String] = None)
  :  RequestT[Identity,Either[ResponseError[io.circe.Error], MyModel], Nothing] = {
    basicRequest
      .get(uri"https://api.github.com/search/repositories?q=$query&sort=$sort")
      .response {
        fromMetadata { meta =>
          meta.code match {
            case StatusCode.Ok => serializer.asJson[ResultModel]
            case StatusCode.BadRequest => serializer.asJson[ErrorModel]
          }
        }
      }
}

[adamw]

@chameleon82 ah yes, that' beacuse ResponseAs was not covariant (see softwaremill/sttp#408). This is fixed in sttp-client 2.0.0-RC11. Maybe you can try that?

[adamw]

And yes, you can abstract over the serialization specifics. In the end, you need to provide a ResponseAs[T, S] value, which describes what to do with the response. How this value is created is arbitrary.

[chameleon82]

@adamw Thanks, code fixed with RC11. But still can't come up with an abstraction. Could you help to provide example?
If it possible to do, then, probably don't need to extend models and client functionality could be enough.

[adamw]

this compiles just fine:

import sttp.client._
import sttp.model.StatusCode

object Test extends App {
  sealed trait MyModel
  case class SuccessModel() extends MyModel
  case class ErrorModel() extends MyModel

  trait SerializerApi {
    def asJson[T]: ResponseAs[Either[ResponseError[io.circe.Error], T], Nothing]
  }

  class RepositoryApi(serializer: SerializerApi) {
    def getRepositories(
        query: String,
        sort: Option[String] = None
    ): RequestT[Identity, Either[ResponseError[io.circe.Error], MyModel], Nothing] = {
      basicRequest
        .get(uri"https://api.github.com/search/repositories?q=$query&sort=$sort")
        .response {
          fromMetadata { meta =>
            meta.code match {
              case StatusCode.Ok         => serializer.asJson[SuccessModel]
              case StatusCode.BadRequest => serializer.asJson[ErrorModel]
            }
          }
        }
    }
  }
}

direct modification of your example :)

[chameleon82]

@adamw I stuck with same solution :)
It want Decoder[T] to be passed when i trying to use implementation, but trait don't have it

trait SerializerApi {
    def asJson[T]: ResponseAs[Either[ResponseError[io.circe.Error], T], Nothing]
  }

I tried next approach:

import io.circe.generic.AutoDerivation
import sttp.client._
import sttp.client.circe.SttpCirceApi
import sttp.model.StatusCode

object Test extends App {

  sealed trait MyModel

  case class SuccessModel() extends MyModel

  case class ErrorModel() extends MyModel

  trait SerializerApi[DECODER[_]] {
    def asJson[T: DECODER : IsOption]: ResponseAs[Either[ResponseError[io.circe.Error], T], Nothing]
  }

  class RepositoryApi[DECODER](serializer: SerializerApi[DECODER]) {
    
    import serializer._  // import autodecoders
    
    def getRepositories(
                         query: String,
                         sort: Option[String] = None
                       ): RequestT[Identity, Either[ResponseError[io.circe.Error], MyModel], Nothing] = {
      basicRequest
        .get(uri"https://api.github.com/search/repositories?q=$query&sort=$sort")
        .response {
          fromMetadata { meta =>
            meta.code match {
              case StatusCode.Ok => serializer.asJson[SuccessModel]
              case StatusCode.BadRequest => serializer.asJson[ErrorModel]
            }
          }
        }
    }
  }

  class CirceSerializer extends SerializerApi[io.circe.Decoder] with SttpCirceApi with AutoDerivation 
  
  val serializer = new CirceSerializer
  val api = new RepositoryApi(serializer)
  implicit val backend = HttpURLConnectionBackend()
  val response = api.getRepositories(query = "http language:scala").send()

}

but it wont compile also due implicit decoders not provided in trait

But it works fine if I change it to specific implementation

  class RepositoryApi(serializer: CirceSerializer) {

Seems issue related to client, not to model. Should i open new issue in client for that?

---

Back to topic i see tapir use jsonBody to specify different code responses ( https://tapir-scala.readthedocs.io/en/latest/endpoint/statuscodes.html#dynamic-status-codes) . I think it make sense to have jsonBody in sttp-model as common dec/enc implementation free abstraction. So example above even related to client still make sense here

[adamw]

Ah you want to abstract json generation ... the problem is that json codec derivation with circe is a compile-time mechanism, to in order to perform the derivation, you need to know (at compile-time) the concrete class for which to derive the codec.

So at the point where you call serializer.asJson[Something], you need to know what typeclass the compiler should derive.

The tapir codecs might end up in a separate project (maybe here as you write), however they are more powerful than what's needed in the client: the additionally capture validation, schema, and are bi-directional. On the other hand, for the client it's enough to decode (for responses) and encode (for requests).

[chameleon82]

Following generic response as type RequestT[Identity, Either[ResponseError[Exception], T], Nothing] what do you suggest for improve next code?

I think generally it is good to return Success(model) for successful (2xx) result and Error/Exception for any other cases. And for expected error cases like not found i can have general ApiError which can be useful to separate business logic errors from transport/serialization errors.
At the same time i want to have api method self-described as much as possible before i do actual call send().

So right now i can return ApiError[ApiResponse] as exception explicitly throwing it:

case class ApiResponse(  code: Option[Int] = None,  message: Option[String] = None) extends ApiModel

case class ApiError[T](val model:T) extends Exception

    basicRequest
      .method(Method.GET, uri"$baseUrl/pet/${petId}")
      .response(
        asJson[Pet].mapWithMetadata{
           case (Right(value),_) => Right(value)
           case (Left(ex: HttpError),responseMetadata) => responseMetadata.code match {
             case StatusCode.NotFound => throw new ApiError(serialization.read[ApiResponse](ex.body))
             case _ => Left(ex)
      }

but ii seems is not to be correct in general as my result type is RequestT[Identity, Either[ResponseError[Exception], T].

Would it better to have an ApiError as part of the ResponseError?

sealed abstract class ResponseError[+T] extends Exception {
  def body: String
}
case class HttpError(body: String) extends ResponseError[Nothing]
case class DeserializationError[T](body: String, error: T) extends ResponseError[T]

case class ApiError[T](body: String, error: T) extends ResponseError[T]

Or my expectations is not really correct and i should operate with more generic RequestT[Identity, Either[Exception, T], Nothing] type?

ps: it could be also useful if HttpError will have status like:
case class HttpError(code: StatusCode, body: String) extends ResponseError[Nothing]

[adamw]

I think it depends on what your target model is. You can either have typed errors - where at least some of the errors are represented as a case class - or untyped errors, just using Exception.

If the latter, then the generic response type should be RequestT[Identity, Either[ResponseError[Exception], T], Nothing] as you write.

If you do have typed errors, then you would probably need a custom hierarchy, covering three cases:

1. non-2xx response, successfully parsed http error
2. non-2xx response, unknown http error
3. unparseable 2xx body

I'd represent transport errors as exceptions (or failed effects), as these are distinct from the errors described above: in that case, there's no response received at all.

There's no need for HttpError to contain the status code as that's not part of the body, but part of the response meta-data. This is always available on the Response type, regardless of how the body is handled.