[ENH]: Return database id in get collections call from sysdb

[sanketkedia]

Description of changes

Summarize the changes made by this PR.

• Improvements & Bug fixes
  • Organizing data of collections in s3 by s3 prefixes requires knowledge about database id in the write path. This PR populates and returns database id from sysdb in get_collections rpc
  • We skip serializing and returning this from FE to client
  • Even though the collection model is the same for both local and distributed, we don't set it in local currently. However, it can be done if needed
• New functionality
  • ...

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes

None

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[sanketkedia]

• [ENH]: Set prefix path in sysdb register #4804
• [ENH]: Support writing data to separate prefixes in s3 #4753
• [ENH]: Plumb prefix path all the way to the bf writer #4743
• [ENH]: Return database id in get collections call from sysdb #4686 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[propel-code-bot]

Expose Database ID in SysDB get_collections and Collection Model for Storage Organization

This PR adds database ID information to the Rust, Go, and Protobuf representations of a Collection, and ensures that SysDB includes the database ID when returning collection metadata (e.g., through get_collections). This allows downstream systems (e.g., storage backends, S3 planners) to organize data by database context, which is particularly necessary for S3 prefix-based storage strategies. The change propagates through the full stack: protobuf definitions, sysdb implementations (SQLite and gRPC), Rust domain types, exposure in API types, test reference models, and relevant conversion/utilities. Compatibility with older sysdb deployments and future-proofing for mixed-version scenarios is addressed in the implementation and test code.

Key Changes:
• Protobuf (chroma.proto): Added optional database_id field to the Collection message.
• Rust types (collection.rs, api_types.rs): Added a newtyped DatabaseUuid to the Collection struct, serialization/deserialization logic, and updated conversions for compatibility with collections lacking a database ID.
• SysDB storage implementation (sqlite.rs, sysdb.rs): SQLite lookups and result rows now fetch and propagate database_id; error handling improved on parsing failures.
• Go sysdb implementation: convertCollectionToProto attaches the database ID field when serializing Collection proto.
• APITypes: Added DatabaseIdParseError and appropriate error handling for parsing/serializing collection-level database IDs.
• Unit/integration tests updated to verify parsing, fallback, and end-to-end propagation of database_id.

Affected Areas:
• SysDB storage and API (both SQLite and gRPC, and Go/Rust backends)
• Collection metadata model in both Rust and Go
• Proto interfaces and cross-service contracts (chroma.proto)
• Test coverage for sysdb and collection parsing

This summary was automatically generated by @propel-code-bot

[HammadB]

discussed offline - maybe handle b/w compat with Optional on database id on collection