Logging your application using Amazon CloudWatch Logs

Posted on by Panther

When developing and running applications, logging is an essential part of monitoring and debugging. Amazon CloudWatch Logs is a service provided by AWS that allows you to monitor, store, and access your log files from Amazon EC2 instances, AWS CloudTrail, and other AWS services.

In this tutorial, we will discuss how to set up and use Amazon CloudWatch Logs to monitor and store log files from your applications. This tutorial assumes that you already have an AWS account and are familiar with basic AWS concepts.

Step 1: Create a CloudWatch Logs group

A log group is a named collection of log streams that share the same retention, monitoring, and access control settings. To create a log group, follow these steps:

  1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.
  2. In the navigation pane, choose “Log groups”.
  3. Choose “Create log group”.
  4. In the “Create log group” dialog box, enter a name for the log group.
  5. Choose “Create log group”.

You have now created a log group, which is a container for your log streams.

Step 2: Create a CloudWatch Logs stream

A log stream is a sequence of log events that share the same source. Each log stream belongs to a single log group. To create a log stream, follow these steps:

  1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.
  2. In the navigation pane, choose “Log groups”.
  3. Choose the log group that you created in Step 1.
  4. Choose “Create log stream”.
  5. In the “Create log stream” dialog box, enter a name for the log stream.
  6. Choose “Create log stream”.

You have now created a log stream, which is a sequence of log events that share the same source.

Step 3: Configure your application to send log data to CloudWatch Logs

There are several ways to send log data to CloudWatch Logs, depending on your application architecture and programming language. Here are some examples:

Using AWS SDK for Java

// Create a CloudWatchLogs client
AWSLogsClientBuilder builder = AWSLogsClientBuilder.standard();
AWSLogs client = builder.build();

// Create a log event
InputLogEvent logEvent = new InputLogEvent().withTimestamp(timestamp).withMessage(message);

// Create a log stream
CreateLogStreamRequest createLogStreamRequest = new CreateLogStreamRequest();
createLogStreamRequest.setLogGroupName(logGroupName);
createLogStreamRequest.setLogStreamName(logStreamName);
client.createLogStream(createLogStreamRequest);

// Put the log event into the log stream
PutLogEventsRequest putLogEventsRequest = new PutLogEventsRequest();
putLogEventsRequest.setLogGroupName(logGroupName);
putLogEventsRequest.setLogStreamName(logStreamName);
putLogEventsRequest.setLogEvents(Arrays.asList(logEvent));
client.putLogEvents(putLogEventsRequest);

Using AWS SDK for Python

# Create a CloudWatchLogs client
logs = boto3.client('logs')

# Create a log event
log_event = {
    'timestamp': timestamp,
    'message': message
}

# Create a log stream
logs.create_log_stream(logGroupName=log_group_name, logStreamName=log_stream_name)

# Put the log event into the log stream
logs.put_log_events(logGroupName=log_group_name, logStreamName=log_stream_name, logEvents=[log_event])

Using AWS SDK for Ruby

# Create a CloudWatchLogs client
logs = Aws::CloudWatchLogs::Client.new

# Create a log event
log_event = {
    timestamp: timestamp,
    message: message
}

# Create a log stream
logs.create_log_stream(log_group_name: log_group_name, log_stream_name: log_stream_name)

# Put the log event into the log stream
logs.put_log_events(log_group_name: log_group_name, log_stream_name: log_stream_name, log_events: [log_event])

Using AWS SDK for .NET

// Create a CloudWatchLogs client
var client = new AmazonCloudWatchLogsClient();

// Create a log event
var logEvent = new InputLogEvent
{
    Timestamp = timestamp,
    Message = message
};

// Create a log stream
client.CreateLogStream(new CreateLogStreamRequest
{
    LogGroupName = logGroupName,
    LogStreamName = logStreamName
});

// Put the log event into the log stream
client.PutLogEvents(new PutLogEventsRequest
{
    LogGroupName = logGroupName,
    LogStreamName = logStreamName,
    LogEvents = new List<InputLogEvent> { logEvent }
});

Choose the appropriate programming language and AWS SDK for your application, and replace the variables with your own values. Once you have configured your application to send log data to CloudWatch Logs, it is time to test it.

Step 4: Test your application logging

To test your application logging, run your application and generate some log data. You can use a test application or modify your production application to add logging statements.

After your application generates some log data, check the CloudWatch Logs console to see if the log data has been received. Follow these steps:

  1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.
  2. In the navigation pane, choose “Log groups”.
  3. Choose the log group that you created in Step 1.
  4. Choose the log stream that you created in Step 2.
  5. In the log stream, you should see the log events that your application generated.

Congratulations! You have successfully configured your application to send log data to CloudWatch Logs and tested it.

Step 5: Set up log metric filters

Log metric filters allow you to extract data from your log events and emit custom metrics based on that data. For example, you can set up a log metric filter to count the number of error messages in your application logs.

To set up a log metric filter, follow these steps:

  1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.
  2. In the navigation pane, choose “Log groups”.
  3. Choose the log group that you created in Step 1.
  4. Choose “Create metric filter”.
  5. In the “Create metric filter” dialog box, enter a name for the metric filter and a filter pattern.
  6. Define the metric filter pattern. For example, the following filter pattern counts the number of error messages in your log events: 
[timestamp,severity="ERROR",...]
  1. Choose “Create filter”.

You have now set up a metric filter that will emit custom metrics based on the data in your log events.

Step 6: Create CloudWatch Alarms

CloudWatch Alarms allow you to set up notifications based on metrics, including the custom metrics that you created in Step 5. For example, you can set up an alarm that notifies you when there are more than 10 error messages in your log events.

To create a CloudWatch alarm, follow these steps:

  1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.
  2. In the navigation pane, choose “Alarms”.
  3. Choose “Create alarm”.
  4. Define the alarm conditions, such as the metric to monitor and the threshold.
  5. Define the alarm actions, such as sending an email notification or triggering an AWS Lambda function.
  6. Choose “Create alarm”.

You have now set up a CloudWatch alarm that will notify you when the conditions of the alarm are met.

Conclusion

In this tutorial, we discussed how to set up and use Amazon CloudWatch Logs to monitor and store log files from your applications. We covered how to create a CloudWatch Logs group and stream, how to configure your application to send log data to CloudWatch Logs, how to test your application logging, how to set up log metric filters, and how to create CloudWatch Alarms.

By using Amazon CloudWatch Logs, you can gain visibility into your application logs and quickly identify and respond to issues before they become critical.

Related Post