OpenSearch / ElasticSearch Addon

[dmathewwws]

Hi,

I am setting up an OpenSearch / ElasticSeach cluster as an addon. It works but I had a couple questions:

• There is a part of the code that expose port 443, I'm not 100% sure what the code does to be honest, so im just checking it isn't doing something silly and insecure. I'm hoping it is just exposing port 443 for the ElasticSearch domain I am creating in the Cloudformation.
• I had to hardcode my private subnet. HARDCODED-SUBNET-ID in the below code. Would you have a suggestion to not have to hardcode this?

    SubnetIds: 
    - 'HARDCODED-SUBNET-ID'

I also tried doing this, but ran into an error in Cloudformation saying "Invalid request provided: You must specify exactly one subnet."

    SubnetIds: !Split [ ',', { 'Fn::ImportValue': !Sub '${App}-${Env}-PrivateSubnets' } ]

Parameters:
  App:
    Type: String
    Description: Your application's name.
  Env:
    Type: String
    Description: The environment name your service, job, or workflow is being deployed to.
  Name:
    Type: String
    Description: The name of the service, job, or workflow being deployed.
Resources:
  # Security group to add the DB to the VPC,
  # and to allow the Fargate containers to talk to DB
  ElasticSecurityGroup:
    Metadata:
      'aws:copilot:description': 'A security group to access the DB cluster'
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: "DB Security Group"
      VpcId: { 'Fn::ImportValue': !Sub '${App}-${Env}-VpcId' }
  # Enable ingress from other ECS services created within the environment.
  ElasticSearchIngress:
    Metadata:
      'aws:copilot:description': 'Allow ingress from containers in my application to the DB cluster'
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      Description: Ingress from Fargate containers
      GroupId: !Ref 'ElasticSecurityGroup'
      IpProtocol: tcp
      FromPort: 443
      ToPort: 443
      SourceSecurityGroupId: { 'Fn::ImportValue': !Sub '${App}-${Env}-EnvironmentSecurityGroup' }
  # The cluster itself.
  ElasticsearchDomain:
    Type: 'AWS::OpenSearchService::Domain'
    Properties:
      AccessPolicies:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            AWS: '*'
          Action:
          - 'es:ESHttp*'
          Resource: !Sub 'arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/opensearch-db/*'
      DomainName: opensearch-db
      EngineVersion: 'OpenSearch_1.0'
      EBSOptions:
        EBSEnabled: true
        VolumeSize: 10
        VolumeType: gp2
      ClusterConfig:
        DedicatedMasterEnabled: false
        InstanceCount: 1
        InstanceType: t3.medium.search
      VPCOptions:
        SecurityGroupIds:
        - !Ref ElasticSecurityGroup
        SubnetIds: 
        - 'HARDCODED-SUBNET-ID'
  ElasticEndpointAddressParam:
    Type: AWS::SSM::Parameter
    Properties:
      Name: !Sub "/copilot/${App}/${Env}/secrets/OPENSEARCH_ENDPOINT"
      Type: String
      Value: !GetAtt ElasticsearchDomain.DomainEndpoint
Outputs:
  ElasticSecurityGroup:
    Description: Security group for DB
    Value: !Ref ElasticSecurityGroup

[iamhopaul123]

Hello @dmathewwws. Just curious did you write the addons template on your own or referred it from elsewhere?

There is a part of the code that expose port 443, I'm not 100% sure what the code does to be honest, so im just checking it isn't doing something silly and insecure. I'm hoping it is just exposing port 443 for the ElasticSearch domain I am creating in the Cloudformation.

Yes you are right it is just to allow ingress for port 443 for the ES. More specifically it allows your ECS service to be able to access through that port, because any output security group will be automatically attached to your ECS service. (see here)

I had to hardcode my private subnet. HARDCODED-SUBNET-ID in the below code. Would you have a suggestion to not have to hardcode this?

You could possibly make use of !Select. For example if you want to select the first private subnet:

    SubnetIds: !Select [0, !Split [ ',', { 'Fn::ImportValue': !Sub '${App}-${Env}-PrivateSubnets' } ] ]

[dmathewwws]

thanks @iamhopaul123 .

I did write this addon template (but based it mainly off Efe's Redis template).

The !select idea you suggested works great.

Just curious though. I did a similar Addon for Postgres and didn't have to pick one subnet. I was able to pass in

DBSubnetGroup:
    Type: AWS::RDS::DBSubnetGroup
    Properties:
      DBSubnetGroupDescription: Group of subnets to place DB into
      SubnetIds: !Split [ ',', { 'Fn::ImportValue': !Sub '${App}-${Env}-PrivateSubnets' } ]
  
# And inside setting up Postgres: # 
  DBInstance:
    Type: AWS::RDS::DBInstance
     ....
     DBSubnetGroupName: !Ref 'DBSubnetGroup'

Is there a reason why there is only 1 subnet limitation for Elasticsearch? I feel like I'm missing the bigger problem that is making me hardcode the first subnet.

[iamhopaul123]

Good morning @dmathewwws. Seems like OpenSearchService requires one subnet ID for each Availability Zone that your domain uses (see here). So for example, you can also provide multiple subnets with different AZs respectively.

However, by default Copilot creates private subnets in different AZs, so that problem shouldn't happen. Then I found this interesting comment. Apparently you would need to do more configuration here in order to enable multiple AZs. Hopefully it would be helpful for your question!

[dmathewwws]

thanks :)

[coopsmoss]

If anyone else is interested, here is what I came up with, with just a couple differences

• Port 80 is easier to work with. Since it's all private anyway there's no reason to use https
• Name the OpenSearchDomain with the environment so that you can have a test and production version
• Use the subnets trick as recommended by @iamhopaul123

Parameters:
  App:
    Type: String
    Description: Your application's name.
  Env:
    Type: String
    Description: The environment name your service, job, or workflow is being deployed to.
  Name:
    Type: String
    Description: The name of the service, job, or workflow being deployed.

Resources:
  # Security group to add OS to the VPC,
  # and to allow the Fargate containers to talk to OS
  OpenSearchSecurityGroup:
    Metadata:
      'aws:copilot:description': 'A security group to access OS'
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: !Sub 'The Security Group for ${Name} to access OpenSearch.'
      VpcId:
        'Fn::ImportValue':
          !Sub '${App}-${Env}-VpcId'
      Tags:
        - Key: Name
          Value: !Sub 'copilot-${App}-${Env}-${Name}-OpenSearch-SecurityGroup'

  # Enable ingress from other ECS services created within the environment.
  OpenSearchIngress:
    Metadata:
      'aws:copilot:description': 'Allow ingress from containers in my application to the OpenSearch cluster'
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      Description: Ingress Security Group from Fargate containers
      GroupId: !Ref 'OpenSearchSecurityGroup'
      IpProtocol: tcp
      FromPort: 80
      ToPort: 80
      SourceSecurityGroupId: { 'Fn::ImportValue': !Sub '${App}-${Env}-EnvironmentSecurityGroup' }
  # The cluster itself.
  OpenSearchDomain:
    Type: 'AWS::OpenSearchService::Domain'
    Properties:
      AccessPolicies:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            AWS: '*'
          Action:
          - 'es:ESHttp*'
          Resource: !Sub 'arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/opensearch-${Env}/*'
      DomainName: !Sub opensearch-${Env}
      EngineVersion: 'OpenSearch_1.1'
      EBSOptions:
        EBSEnabled: true
        VolumeSize: 10
        VolumeType: gp2
      ClusterConfig:
        DedicatedMasterEnabled: false
        InstanceCount: 2
        InstanceType: c6g.large.search # probably overkill
      VPCOptions:
        SecurityGroupIds:
        - !Ref OpenSearchSecurityGroup
        SubnetIds:
          - !Select [0, !Split [ ',', { 'Fn::ImportValue': !Sub '${App}-${Env}-PrivateSubnets' } ] ]
      Tags:
        - Key: Name
          Value: !Sub 'copilot-${App}-${Env}-${Name}-OpenSearch-Domain'
  OpenSearchEndpointAddressParam:
    Type: AWS::SSM::Parameter
    Properties:
      Name: !Sub "/copilot/${App}/${Env}/secrets/OPENSEARCH_ENDPOINT"
      Type: String
      Value: !GetAtt OpenSearchDomain.DomainEndpoint
Outputs:
  OpenSearchSecurityGroup:
    Description: Security group for OpenSearch
    Value: !Ref OpenSearchSecurityGroup

And then in your manifest.yml you can do this (assuming your environments are 'test' and 'prod'):

variables:
  OPENSEARCH_PORT: "80"

environments:
  prod:
    secrets: # these correspond to the property in the addon
      OPENSEARCH_ENDPOINT: /copilot/api/prod/secrets/OPENSEARCH_ENDPOINT
  test:
    secrets:
      OPENSEARCH_ENDPOINT: /copilot/api/test/secrets/OPENSEARCH_ENDPOINT

[efekarakus]

Awesome, thanks for sharing @coopsmoss!

[doneumark]

@coopsmoss @efekarakus - so just so I get it right, the steps to do for adding a opensearch service to the app are:

1. create an YML file at "copilot/APP_NAME/addons".
2. paste the content you wrote.
3. add OPENSEARCH_PORT and OPENSEARCH_ENDPOINT to the manifest (do you need to "copilot secret init" first? and if so - to what values?)
4. deploy.

and the fun should begin?
Because I get: "deploy service node to environment test: deploy service: stack api-test-node did not complete successfully and exited with status UPDATE_ROLLBACK_COMPLETE" when trying to deploy.
Should I do anything else before making those changes?
thanks!

[iamhopaul123]

Hello @doneumark. Could you send more detailed error msg about the deploy failure? Like the reason why it fails should be able to found in deployment progress tracker. Also you can access it through the CloudFormation console.

do you need to "copilot secret init" first? and if so - to what values?

You don't need to do that for OPENSEARCH_PORT but OPENSEARCH_ENDPOINT i believe it is required to run copilot secret init to create the secret first. And the value should be your open search endpoint.

[doneumark]

Hi @iamhopaul123, this is the only message I can find about the subject (from the cloudformation stack):
Embedded stack "..." was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [OpenSearchDomain].

[iamhopaul123]

Hello @doneumark. Sorry about the fact that we don't show a full error msg for addons resource creation. For now, I think if you go into the nested stack, you'll be able to see the reason why "OpenSearchDomain" in the nested stack fails to create.