User Management Design

[PhiRho]

Design the correct way to manage users in the amazee.ai system, to account for billing and other integrations.

[PhiRho]

Step one in sorting out the design for the user management is to understand who the users are, and what they need to be able to do/what role they fill. We need user management for two main purposes.

1. To allow amazee.ai to implement usage based billing for calls to underlying AI models.
2. To allow amazee.ai customers to include some form of user management in their own systems.

[PhiRho]

A first pass at defining the personas we need to account for in the system, bearing in mind the common factors between different billing systems, as well as existing concepts in LiteLLM.

Personas

Administrator 📋

An entity outside of the billing flow who is responsible for managing the other personas, either directly or indirectly. The administrator sets up subscriptions, meters, and billable events. They define regions and maintain the LiteLLM proxy.

Customer 💳

The entity responsible for the payment of bills and/or invoices. Every call to the downstream LLM MUST include a reference (either direct or indirect) to the customer_id so that subscriptions and meters can be correctly referenced.
A customer with no associated Private AI Keys cannot incur a bill (as they cannot make authenticated LLM calls). As such there is no need for the customer to be associated with an entity in LiteLLM until such time as a key is created. At that time the user and customer will be associated with an appropriate entity in LiteLLM.

User 👤

The entity making LLM calls via DrupalAI or some other interface. Every user MUST be associated with a Customer, in order to roll up usage and billing correctly. Every customer can have a default user which can be used without any new users being created. One customer may have many users, but each user must have exactly one customer.

Private AI Key 🗝️

The 'entity' which has permissions to call LiteLLM and the underlying models. The private_ai_key is used for both AuthN and AuthZ when calling through to the LLMs. Every call to LiteLLM MUST include a private_ai_key. A user may have any number of keys, but every key must be associated with exactly one user. A private_ai_key MAY suffice as indirect reference to a customer_id for billing systems.

[bonus414]

@PhiRho
This looks good. The other personas I am thinking about (which might be "not just yet" step) are along the lines of

Team Lead/Manager

• Has permission to manage other Users within their team
• Can view team-level usage metrics
• Can allocate resources within Customer-defined limits
• Still belongs to exactly one Customer

This could be just be a role or permission set assigned to certain Users rather than a completely separate entity in your data model but I'm not entirely sure and would want to discuss.

Do you want me to go this granular at this point? I have some other roles/personas I was thinking about.

[PhiRho]

@bonus414

Let's get them all on the page, then we can refine "right now" and "just now" options once we start looking deeper. Knowing that users might need different roles could mean saving them differently, and DB migrations are a pain at the best of times.

[PhiRho]

Adding to the personas. We need a Team (or Group)

Team 👥

A group of users within the scope of a single customer. These users have common permissions, a shared (or common?) budget, and generate a billing/usage report which is a portion of the total customer bill. A user may be a member of multiple teams, but then calls to the downstream LLM MUST reference the team directly for correct flow of billing data (or risk the same usage being metered twice). A team lead may have many teams, and a team may have multiple leads.

---

And we can also add some roles for users

Team Lead/Administrator

A team lead has the ability to manage other users, view billing and usage details, and assign limits within the constraints of the customer. A team lead may have management permissions over all users, or some users, as defined by the customer. Groups of users manged by a team lead are called a Team.

Key Creator

The Key Creator role allows a user to generate, modify, and delete private AI keys associated with their own user. They may view billing and usage reports associate with their own usage.

Read Only

The read only role allows a user to use keys which have been created on their behalf by a team lead, but they may not modify or delete the keys, and cannot generate new keys. They may view billing and usage reports associated with their own usage.

---

Once we have one set of roles in the system, we should be able to extend that to add others as they arise. The AuthZ will be implemented to check roles already. The default role for new users can be set at the Customer level, or the Administrator level.

[PhiRho]

and generate a billing/usage report which is a portion of the total customer bill.

I'm not entirely sure about this piece - it may depend on the implementation within liteLLM, as well as the implementation of the specific solution chosen.

[PhiRho]

@bonus414 - Already with this set of personas I am seeing two distinct tracks of "use cases" for the users and roles. There is the "financial" track which is around the billing and metering, and then there is user and key management for things like separating a dev stack from a prod stack.

Based on experience with the existing customer base and user feedback, are there further tracks I should be considering?

[PhiRho]

I'm going through what some of the needed APIs/endpoints would be, and that leads to another question. Do we want it to be possible for a user to be a member of multiple teams/groups, or at most one?
In terms of team based restrictions multiple teams would mean needing to do the union of rules across all memberships to define rules for a user. If we want to role up metering/usage by team it would also add the risk of double counting.

[PhiRho]

Currently we allow users in amazee.ai to have keys referencing multiple regions (and thus multiple LiteLLM instances). Is this a quality we wish to maintain moving forward, or would it make sense to restrict a user to a single region?

[bonus414]

@PhiRho _Already with this set of personas I am seeing two distinct tracks of "use cases" for the users and roles. There is the "financial" track which is around the billing and metering, and then there is user and key management for things like separating a dev stack from a prod stack.

Based on experience with the existing customer base and user feedback, are there further tracks I should be considering?_

My out loud thoughts here is that

1. i haven't received any indication or evidence for further tracks and this time
2. we focus on these two for now and as beta progresses, the data and feedback will inform us if we need to consider other tracks

[bonus414]

@PhiRho

I'm going through what some of the needed APIs/endpoints would be, and that leads to another question. Do we want it to be possible for a user to be a member of multiple teams/groups, or at most one? In terms of team based restrictions multiple teams would mean needing to do the union of rules across all memberships to define rules for a user. If we want to role up metering/usage by team it would also add the risk of double counting.

If we are (I think I'm conflating terms so I'll go back through this and make some mind maps) thinking of it like this:

Organization = Customer (billing entity)

• Can have multiple teams
• Is responsible for payment
• Sets organization-wide policies

Team

• Belongs to exactly one organization/customer
• Can have multiple users
• Might have team-specific settings or resource allocations

User

• Associated with exactly one organization/customer for billing purposes
• Can potentially belong to multiple teams within that organization
• Uses the system and makes API calls

[bonus414]

@PhiRho

Currently we allow users in amazee.ai to have keys referencing multiple regions (and thus multiple LiteLLM instances). Is this a quality we wish to maintain moving forward, or would it make sense to restrict a user to a single region?

This is one we might need to talk through more. In the Drupal ecosystem a customer/org might have hundreds of sites. Some of these sites might be in different regions and thus they need multiple keys as a result (if I'm getting this right) and if that's the case, I don't know if a user should also be in multiple regions. it's going to depend on who is a site admin or whatever access is allowed on the drupal side for the user.

Like i said, i think we need to think this one through a bit more or i need to understand some things more deeply

[PhiRho]

I think this is the way I view the hierarchy - Keys are restricted to a specific region since they are generated from a specific LiteLLM instance, other than that restrictions should be defined by the customer (who may want to restrict a user to a specific region).