Skip to content

EES-5874 Create Search Function App #5630

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 4, 2025
Merged

Conversation

leeoades
Copy link
Collaborator

New Azure Function App for Search

This PR contains a new function app created to handle new search processes.

Currently, there is only a single Azure Function called CreateSearchableReleaseDocument.

CreateSearchableReleaseDocument Function

This Azure Function is triggered by a new message on a source queue which contains details of a new Release version being published.
It consumes the message and makes a call to the ContentAPI to retrieve a ReleaseSearchViewModel. This contains a searchable HTML version of the release, plus associated metadata.
The function then creates a new Blob in the searchable documents container with this information.
Finally, the function publishes a message about the new Searchable Document to an output queue.

@leeoades leeoades force-pushed the EES-5874-Create-processor branch 2 times, most recently from 0a97142 to 332fc80 Compare February 26, 2025 14:56
@leeoades leeoades requested a review from benoutram February 26, 2025 14:56
@leeoades leeoades force-pushed the EES-5874-Create-processor branch from 332fc80 to bc6db15 Compare February 26, 2025 14:59
throw new UnableToGetPublicationLatestReleaseSearchViewModelException(publicationSlug, e.StatusCode, e.Message);
}

var result = await response.Content.ReadFromJsonAsync<ReleaseSearchViewModelDto>(cancellationToken: cancellationToken);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just letting you know there could be a problem here as I think Content API responses are serialised using Newtonsoft.Json, but this is using System.Text.Json. They are not always compatible but it might just be the fields involved here are working fine. I seem to remember there's a particular incompatibility around enums and this has been a problem for the Public API where the Admin (which also uses Newtonsoft.Json) consumes data from the Public API which uses System.Text.Json.

We want to migrate away from Newtonsoft.Json in favour of System.Text.Json, but we haven't completed this yet.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That will be a configuration issue then. I'm hoping/praying that you're serialising the enums as their string label and not as some arbitrary number? There should be no compatibility issues as json serialisation is well defined and both implementations follow it. I'll double check to ensure the view model the API is producing is outputting the string.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I've just checked and the view model does not have any enums on it.
I've also noticed your EnumToEnumValueJsonConverter. I'm sure this is a configuration setting and that we can set that instead. Let me see if I can find it...

Copy link
Collaborator

@benoutram benoutram Mar 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, mostly configuration issues that can either be resolved with out-of-the-box supported config or other workarounds.

There's a documented list of differences here in case you've not already seen it.

I think with enums it's easy to just add in the System.Text.Json string value converter, see enums-as-strings.

Our own EnumToEnumValueJsonConverter is doing something different. Thanks for pointing it out because that's where the incompatibility around enums came from for the Public API that I was thinking of, where we've also had to write our own System.Text.Json variant of that converter. There's also EnumToEnumLabelJsonConverter and the System.Text.Json variant of that.

'EnumToEnumValue' and 'EnumToEnumLabel' refer to getting the label or value from an EnumLabelValueAttribute annotated enum entry. E.g. GeographicLevel.LocalAuthority is annotated with a label of 'LocalAuthority' and value of 'LA'. With hindsight I think we probably regret this custom implementation! 😂

[EnumLabelValue("Local authority", "LA")]
LocalAuthority,

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yikes. Well, for the view model that I am consuming, there are no such gremlins. And if we were to add an enum value to view model, I would represent it as a string through an explicit conversion and not rely on the serialiser. The dto can be unit tested also to ensure that the json is deserialised correctly.

@leeoades leeoades force-pushed the EES-5874-Create-processor branch 2 times, most recently from 376af36 to d3b73f7 Compare March 3, 2025 16:33
@benoutram benoutram self-requested a review March 4, 2025 16:07
…est release with the associated metadata

Move ReleaseSearchViewModel to own file. Code tidy.
WIP Putting together the new Content Processor for Search
Rename releaseId to releaseVersionId

rename releaseId to releaseVersionId in Content Processor

code tidy

Ensure the summary is made safe for upload

Ensure Title is also made ascii and no spaces. Also, replace the magic strings

Code tidy

Add Azure Blob Storage Client integration tests. Also added a blob download and delete methods.

Add a couple more tests for Download and Delete non-existent blobs

Add tests for HostBuilder. Flow cancellation token through everywhere. Add tests for SearchableDocumentCreator

Rename the search Processor to Search.FunctionApp

Moved projects to Search/Function App solution folder

Mask connection string

Code tidy

Getting the function app set up

Test config and container registrations set up

If service is not configured, fail on startup instead of waiting for a message to arrive

Fix config

Severed the ref to viewmodels. Added dto. Code Tidy. Removed unused refs.

Remove the hardcoded queue names from the trigger function and move them to config

Add the local.settings.json file

Rename the function

Add ReleaseId to the searchable view model to use as the blob name

Fix

Fix
Fix merge

Amendments made based on PR feedback

Moved Azure Storage code that was only being used by the integration tests into the test project to clean up the client.

Add Search Function App to the build yaml

Add search function app to the start.ts useful scripts

Remove config for durable storage as we have gone with the non-durable approach

Make the metadata keys ALL-CAPS-KEBABS. Also, rename Type to ReleaseType for added clarity.
@leeoades leeoades force-pushed the EES-5874-Create-processor branch from bc921b1 to 8056d78 Compare March 4, 2025 16:09
@leeoades leeoades merged commit 2da25bc into dev Mar 4, 2025
11 checks passed
@leeoades leeoades deleted the EES-5874-Create-processor branch March 4, 2025 16:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants