Amazon CloudWatch

Kurrent Cloud supports integration with Amazon CloudWatch for both logs and metrics. Below you'll find instructions on how to set up both.

Info

The Amazon CloudWatch integration is currently in closed beta and may change over time. To access it, please contact support.

Logs sink

Amazon CloudWatch is a monitoring service. The "logs" portion of its functionality is separated into it's own integration in Kurrent Cloud for ease of use.

You can cause all the logs generated by each KurrentDB running in your clusters by creating a custom Amazon CloudWatch log group, an IAM user, and setting up an integration as described below.

Note

If you're using Terraform, instructions on how to perform the steps below can be found here.

Create a CloudWatch Log group

In the AWS console, go to the CloudWatch page.

Make sure the region you'd like to use is selected in the upper right corner (this will ideally be the same region used by the Kurrent Cloud clusters you'll be using as the source).

In the left sidebar, expand Logs and click Log groups. Then click on the button labeled Create log group.

Enter a log group. If you have multiple projects it could be a good idea to include it in the name, such as EventStoreLogs-Production.

Select a retention policy depending on your needs, any tags if necessary, and then click create.

Now find the new log group in the list, and click on it. The next page will show you it's details, including the ARN. Copy it for the next step.

Create IAM credentials

Note

You should create dedicated credentials for use with the Kurrent Cloud integration that only have access to the CloudWatch Log group, as shown here.

In the AWS console, go to the IAM homepage.

In the sidebar, click on Users, and then on the button labeled Add users.

Enter an appropriate name, select the AWS credential type Access key, and then click the next button.

Under set permissions, select "Attach existing policies directly". Don't click on "create policy".

Now click the next button until you get to the page titled "Review." Give the policy an appropriate name and click "Create user".

At this point copy the access key ID and secret access key as you'll need them later. Then click "close"

You should now see the users list again. Click on the newly created user.

Under the Permissions table click "Add inline policy".

On the "Create Policy" screen, click the tab labeled JSON, and then enter the following:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "logs:DescribeLogGroups"
      ],
      "Effect": "Allow",
      "Resource": "${DescribeARN}"
    },
    {
      "Action": [
        "logs:CreateLogStream",
        "logs:DescribeLogStreams",
        "logs:PutLogEvents"
      ],
      "Effect": "Allow",
      "Resource": "${ARN}:*"
    }
  ]
}

Now make two replacements:

Replace ${ARN} with the ARN you copied when you create the CloudWatch Log group in the previous step
For ${DescribeARN}, take the ARN and remove the last part before the asterisk. So for example, if the ARN is arn:aws:logs:us-west-2:123456789012:log-group:EventStoreLogs-Production:*" you'll want this value to be arn:aws:logs:us-west-2:123456789012:log-group:*".

Click on Review policy. As you can see, the recommended policy will ultimately give the integration permission to describe the log groups in your account and write to the newly create log group.

Give the policy some appropriate name, and click on "Create policy".

Add a new integration

In the Kurrent Cloud console, select an organization and then a project.
Once viewing a project, you should see Integrations under the heading Project in the sidebar to the left. Click it.
Click New Integration. Enter a name that will make it easier to find later, and then select Logs. Next select the Sink AwsCloudWatchLogs as in the screen below.
Under Configuration, by Group Name enter the name of the CloudWatch group you created earlier (for example, EventStoreLogs-Production).
Under Configuration, by Aws Region enter the AWS region of the CloudWatch group you created earlier.
Under Configuration, by Clusters select one or more clusters whose logs you want sent to the CloudWatch group.
Under Credentials enter Access Key Id which was shown when you created the IAM user earlier.
Under Credentials enter Secret Access Key which was shown when you created the IAM user earlier.

Testing the integration

Log integration with Kurrent Cloud will take a few minutes to be fully active.

If you're making sure the integration is set up correctly it's a good idea to write a few test events to the cluster. Clusters which are seeing no activity (such as dev or test clusters) will sometimes emit no logs at all.

Additionally, the Kurrent Cloud lacks a way to assert if the IAM credentials are valid. If no new logs appears in your Amazon CloudWatch group, please double-check that the credentials given to the integration are correct and that the IAM user has the appropriate permissions.

Metrics sink

Amazon CloudWatch is a monitoring service. The "metrics" portion of its functionality is separated into its own integration in Kurrent Cloud for ease of use.

You can cause all the metrics generated by each KurrentDB running in your clusters by creating a custom IAM user and setting up an integration as described below.

Note

If you're using Terraform, instructions on how to perform the steps below can be found here.

Create new IAM resources

Note

You should create credentials especially for use with the Kurrent Cloud integration which only have permission to perform the cloudwatch:PutMetricData action on a small set of namespaces, as shown here.

First, decide what namespace you want your KurrentDB metrics to appear under. A decent choice could be simply "EventStoreCloud".

In the AWS console, go to the IAM homepage.

In the sidebar, click on Users, and then on the button labeled Add users.

Enter an appropriate name, select the AWS credential type Access key, and then click the next button.

Under set permissions, select "Attach existing policies directly". Don't click on "create policy".

Now click the next button until you get to the page titled "Review." Give the policy an appropriate name and click "Create user".

At this point copy the access key ID and secret access key as you'll need them later. Then click "close"

You should now see the users list again. Click on the newly created user.

Under the Permissions table click "Add inline policy".

On the "Create Policy" screen, click the tab labeled JSON, and then enter the following:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "cloudwatch:PutMetricData"
      ],
      "Effect": "Allow",
      "Resource": "*",
      "Condition": {
          "ForAnyValue:StringEqualsIgnoreCase": {
                "cloudwatch:namespace": [
                    "EventStoreCloud",
                    "EventStoreCloud/KurrentDB",
                    "EventStoreCloud/host"
                ]
            }
        }
    }
  ]
}

If you want to use a namespace other than EventStoreCloud, simply replace it in the policy above.

Click on Review policy.

Give the policy some appropriate name, and click on "Create policy".

Add a new integration

In the Kurrent Cloud console, select an organization and then a project.
Once viewing a project, you should see Integrations under the heading Project in the sidebar to the left. Click it.
Click New Integration. Enter a name that will make it easier to find later, and then select Logs. Next select the Sink AwsCloudWatchMetric as in the screen below.
Under Configuration, by Namespace enter the namespace you decided on earlier (for example, EventStoreLogs-Production).
Under Configuration, by Aws Region enter the AWS region that matches the clusters you'll be pulling from.
Under Configuration, by Clusters select one or more clusters whose metrics you want sent to CloudWatch.
Under Credentials enter Access Key Id which was shown when you created the IAM user earlier.
Under Credentials enter Secret Access Key which was shown when you created the IAM user earlier.

Testing the integration

Metrics integration with Kurrent Cloud will take a few minutes to be fully activated.

Metrics will typically appear after that without any action on your part.

Additionally, the Kurrent Cloud lacks a way to assert if the IAM credentials are valid. If no metrics appear under the given namespace in CloudWatch, please double-check that the credentials given to the integration are correct and that the IAM user has the appropriate permissions.