Skip to content

Python: .Net: MEVD Preview 2 APR 2025 #11028

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 87 commits into from
Apr 29, 2025
Merged

Python: .Net: MEVD Preview 2 APR 2025 #11028

merged 87 commits into from
Apr 29, 2025

Conversation

westey-m
Copy link
Contributor

Motivation and Context

Description

Contribution Checklist

roji and others added 19 commits January 24, 2025 20:29
Implement LINQ-based vector search filtering
    
Closes #10156
Does most of #10194
And the obsoleted Filter to OldFilter. Continues #10273.
### Description

Merging changes from main and resolving merge conflicts

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄

---------

Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Vincent Biret <[email protected]>
Co-authored-by: Evan Mattson <[email protected]>
Co-authored-by: Dmytro Struk <[email protected]>
Co-authored-by: Chris <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tao Chen <[email protected]>
Co-authored-by: Roger Barreto <[email protected]>
Co-authored-by: Mark Wallace <[email protected]>
Co-authored-by: Estefanía Tenorio <[email protected]>
Co-authored-by: davidpene <[email protected]>
Co-authored-by: ThDuquennoy <[email protected]>
Co-authored-by: Thomas DUQUENNOY <[email protected]>
Co-authored-by: Eduard van Valkenburg <[email protected]>
Co-authored-by: Evan Mattson <[email protected]>
Co-authored-by: Rob Emanuele <[email protected]>
Co-authored-by: K. Andrew Parker <[email protected]>
Co-authored-by: SergeyMenshykh <[email protected]>
Co-authored-by: jenfoxbot <[email protected]>
Co-authored-by: Ben Thomas <[email protected]>
Co-authored-by: Ben Thomas <[email protected]>
Co-authored-by: Adit Sheth <[email protected]>
Co-authored-by: Adit Sheth <[email protected]>
Co-authored-by: ふぁー <[email protected]>
Co-authored-by: Vincent Biret <[email protected]>
Co-authored-by: David A. Torres <[email protected]>
Co-authored-by: Genevieve Warren <[email protected]>
Co-authored-by: Atiqur Rahman Foyshal <[email protected]>
Co-authored-by: Md. Atiqur Rahman Foyshal <[email protected]>
Co-authored-by: Fabian Williams <[email protected]>
Co-authored-by: [email protected] <[email protected]>
Co-authored-by: Ram.Type-0 <[email protected]>
Co-authored-by: Tommy Falgout <[email protected]>
Co-authored-by: Gary Tang <[email protected]>
Co-authored-by: Eirik Tsarpalis <[email protected]>
Co-authored-by: Tommaso Stocchi <[email protected]>
Co-authored-by: Chris Rickman <[email protected]>
Co-authored-by: Devis Lucato <[email protected]>
Co-authored-by: K. Andrew Parker <[email protected]>
Co-authored-by: Jose Luis Latorre Millas <[email protected]>
Co-authored-by: Carsten Lemm <[email protected]>
Co-authored-by: Stephen Toub <[email protected]>
Co-authored-by: Musale Martin <[email protected]>
Co-authored-by: Ross Smith <[email protected]>
### Motivation and Context

Adding hybrid search to March release branch

### Description

Adding hybrid search to March release branch

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
fixes #10416

---------

Co-authored-by: westey <[email protected]>
Co-authored-by: Shay Rojansky <[email protected]>
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

Related: #10596

This PR enhances the vector store functionality by adding builder
pattern implementations and logging decorators for key vector data
interfaces, along with supporting utilities and DI integration.

Interfaces handled:
- `IKeywordHybridSearch<TRecord>`
- `IVectorizableTextSearch<TRecord>`
- `IVectorizedSearch<TRecord>`
- `IVectorStore`
- `IVectorStoreRecordCollection<TKey, TRecord>`

Class types added:
- Builders: implement a pipeline pattern with `Use` and `Build` methods.
- Logging Extensions: provide reusable logging methods for tasks and
enumerables, handling success, failure and cancellation.
- Logging Decorators: wrap the interfaces to add logging by using
`LoggingExtensions` reusable logging methods.
- Builder Extensions: Add `AsBuilder` methods to convert service
instances into builders.
- Logging Builder Extensions: Add `UseLogging` methods to integrate
logging into builder pipelines.
- Service Collection Extensions: Add `Add{T}` and `AddKeyed{T}` methods
for DI registration with configurable lifetimes.

Other changes:
- Split `Verify` class into `Verify` and `KernelVerify` in order to be
able to use `Verify` methods in `Microsoft.Extensions.VectorData`
package without a reference to Semantic Kernel specific logic, like
`ValidPluginName`, `ValidFunctionName` etc.
- Added unit tests for new classes and methods.
- Added usage example in `Concepts/Memory` folder.

### Usage example
Logging with manual registration:
```csharp
var vectorStore = new InMemoryVectorStore()
    .AsBuilder()
    .UseLogging(this.LoggerFactory)
    .Build();
```

Logging with DI:
```csharp
serviceCollection.AddInMemoryVectorStore();

serviceCollection
    .AddVectorStore(s => s.GetRequiredService<InMemoryVectorStore>())
    .UseLogging(this.LoggerFactory);
```

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
@markwallace-microsoft markwallace-microsoft added .NET Issue or Pull requests regarding .NET code kernel Issues or pull requests impacting the core kernel memory labels Mar 18, 2025
@github-actions github-actions bot changed the title MEVD Preview VNext (DRAFT) .Net: MEVD Preview VNext (DRAFT) Mar 18, 2025
### Motivation and Context

Fix AzureAISearch tests and two small bugs.
Improve MongoDB tests with a small fix too.
Fix CosmosNoSQL integration tests
Fixes for postgres and mongodb
Fixes for redis

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
@westey-m westey-m changed the title Python: .Net: MEVD Preview VNext (DRAFT) Python: .Net: MEVD Preview 2 APR 2025 Apr 28, 2025
@westey-m westey-m marked this pull request as ready for review April 28, 2025 21:41
@westey-m westey-m requested a review from a team as a code owner April 28, 2025 21:41
### Motivation and Context

We changed the storage type of the InMemory store to contain a wrapper
with both
the original record and generated vectors, and this code was returning
the wrapper
instead of the original record on get.

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
### Motivation and Context

We have some broken links in the readme's and they are still referring
to memory store which is being replaced.

### Description

- Updated readme files to not advertise memory store anymore.
- Updated readme files to point to learn site docs.

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
### Motivation and Context

The docker container for mongo db is often not accessible when running
the tests

### Description

- Switching to the mongo db test container to try and reduce the
flakiness of the tests
- Adding retries

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
@westey-m westey-m added this pull request to the merge queue Apr 29, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 29, 2025
@westey-m westey-m added this pull request to the merge queue Apr 29, 2025
Merged via the queue into main with commit 68b11d6 Apr 29, 2025
20 checks passed
@westey-m westey-m deleted the feature-vector-data-preb2 branch April 29, 2025 15:43
@westey-m westey-m restored the feature-vector-data-preb2 branch April 29, 2025 19:21
mccoylstevens pushed a commit to mccoylstevens/semantic-kernel that referenced this pull request May 2, 2025
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄

---------

Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Vincent Biret <[email protected]>
Co-authored-by: Shay Rojansky <[email protected]>
Co-authored-by: Evan Mattson <[email protected]>
Co-authored-by: Dmytro Struk <[email protected]>
Co-authored-by: Chris <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tao Chen <[email protected]>
Co-authored-by: Roger Barreto <[email protected]>
Co-authored-by: Mark Wallace <[email protected]>
Co-authored-by: Estefanía Tenorio <[email protected]>
Co-authored-by: davidpene <[email protected]>
Co-authored-by: ThDuquennoy <[email protected]>
Co-authored-by: Thomas DUQUENNOY <[email protected]>
Co-authored-by: Eduard van Valkenburg <[email protected]>
Co-authored-by: Evan Mattson <[email protected]>
Co-authored-by: Rob Emanuele <[email protected]>
Co-authored-by: K. Andrew Parker <[email protected]>
Co-authored-by: SergeyMenshykh <[email protected]>
Co-authored-by: jenfoxbot <[email protected]>
Co-authored-by: Ben Thomas <[email protected]>
Co-authored-by: Ben Thomas <[email protected]>
Co-authored-by: Adit Sheth <[email protected]>
Co-authored-by: Adit Sheth <[email protected]>
Co-authored-by: ふぁー <[email protected]>
Co-authored-by: Vincent Biret <[email protected]>
Co-authored-by: David A. Torres <[email protected]>
Co-authored-by: Genevieve Warren <[email protected]>
Co-authored-by: Atiqur Rahman Foyshal <[email protected]>
Co-authored-by: Md. Atiqur Rahman Foyshal <[email protected]>
Co-authored-by: Fabian Williams <[email protected]>
Co-authored-by: [email protected] <[email protected]>
Co-authored-by: Ram.Type-0 <[email protected]>
Co-authored-by: Tommy Falgout <[email protected]>
Co-authored-by: Gary Tang <[email protected]>
Co-authored-by: Eirik Tsarpalis <[email protected]>
Co-authored-by: Tommaso Stocchi <[email protected]>
Co-authored-by: Chris Rickman <[email protected]>
Co-authored-by: Devis Lucato <[email protected]>
Co-authored-by: K. Andrew Parker <[email protected]>
Co-authored-by: Jose Luis Latorre Millas <[email protected]>
Co-authored-by: Carsten Lemm <[email protected]>
Co-authored-by: Stephen Toub <[email protected]>
Co-authored-by: Musale Martin <[email protected]>
Co-authored-by: Ross Smith <[email protected]>
Co-authored-by: Adam Sitnik <[email protected]>
Co-authored-by: Rodrigo Martins Racanicci <[email protected]>
Co-authored-by: unsafecode <[email protected]>
Co-authored-by: QuocDatHoang <[email protected]>
Co-authored-by: Dat Hoang Quoc <[email protected]>
Co-authored-by: Damien Guard <[email protected]>
glorious-beard pushed a commit to glorious-beard/semantic-kernel that referenced this pull request May 6, 2025
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄

---------

Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Vincent Biret <[email protected]>
Co-authored-by: Shay Rojansky <[email protected]>
Co-authored-by: Evan Mattson <[email protected]>
Co-authored-by: Dmytro Struk <[email protected]>
Co-authored-by: Chris <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tao Chen <[email protected]>
Co-authored-by: Roger Barreto <[email protected]>
Co-authored-by: Mark Wallace <[email protected]>
Co-authored-by: Estefanía Tenorio <[email protected]>
Co-authored-by: davidpene <[email protected]>
Co-authored-by: ThDuquennoy <[email protected]>
Co-authored-by: Thomas DUQUENNOY <[email protected]>
Co-authored-by: Eduard van Valkenburg <[email protected]>
Co-authored-by: Evan Mattson <[email protected]>
Co-authored-by: Rob Emanuele <[email protected]>
Co-authored-by: K. Andrew Parker <[email protected]>
Co-authored-by: SergeyMenshykh <[email protected]>
Co-authored-by: jenfoxbot <[email protected]>
Co-authored-by: Ben Thomas <[email protected]>
Co-authored-by: Ben Thomas <[email protected]>
Co-authored-by: Adit Sheth <[email protected]>
Co-authored-by: Adit Sheth <[email protected]>
Co-authored-by: ふぁー <[email protected]>
Co-authored-by: Vincent Biret <[email protected]>
Co-authored-by: David A. Torres <[email protected]>
Co-authored-by: Genevieve Warren <[email protected]>
Co-authored-by: Atiqur Rahman Foyshal <[email protected]>
Co-authored-by: Md. Atiqur Rahman Foyshal <[email protected]>
Co-authored-by: Fabian Williams <[email protected]>
Co-authored-by: [email protected] <[email protected]>
Co-authored-by: Ram.Type-0 <[email protected]>
Co-authored-by: Tommy Falgout <[email protected]>
Co-authored-by: Gary Tang <[email protected]>
Co-authored-by: Eirik Tsarpalis <[email protected]>
Co-authored-by: Tommaso Stocchi <[email protected]>
Co-authored-by: Chris Rickman <[email protected]>
Co-authored-by: Devis Lucato <[email protected]>
Co-authored-by: K. Andrew Parker <[email protected]>
Co-authored-by: Jose Luis Latorre Millas <[email protected]>
Co-authored-by: Carsten Lemm <[email protected]>
Co-authored-by: Stephen Toub <[email protected]>
Co-authored-by: Musale Martin <[email protected]>
Co-authored-by: Ross Smith <[email protected]>
Co-authored-by: Adam Sitnik <[email protected]>
Co-authored-by: Rodrigo Martins Racanicci <[email protected]>
Co-authored-by: unsafecode <[email protected]>
Co-authored-by: QuocDatHoang <[email protected]>
Co-authored-by: Dat Hoang Quoc <[email protected]>
Co-authored-by: Damien Guard <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation kernel.core kernel Issues or pull requests impacting the core kernel memory .NET Issue or Pull requests regarding .NET code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants