Appearance

AWS CloudTrail ​

Datable integrates with AWS CloudTrail to collect and analyze AWS API activity logs, providing visibility into user actions, resource changes, and security events across your AWS infrastructure.

How it works ​

The CloudTrail integration connects to your S3 bucket containing CloudTrail logs and processes them in real-time. As new log files are delivered to S3, Datable automatically ingests and parses them for analysis.

Prerequisites ​

  • An active AWS CloudTrail trail
  • CloudTrail logs delivered to an S3 bucket
  • AWS IAM credentials with appropriate permissions

Setup Instructions ​

Step 1: Ensure CloudTrail is Configured ​

  1. Sign in to the AWS Management Console
  2. Navigate to CloudTrail
  3. Verify you have an active trail or create a new one
  4. Note the S3 bucket name where logs are delivered

Step 2: Configure S3 Event Notifications ​

For real-time log processing:

  1. Navigate to your CloudTrail S3 bucket
  2. Go to Properties → Event notifications
  3. Create a new event notification:
    • Event types: All object create events
    • Destination: SQS Queue (will be provided by Datable)

Step 3: Configure the Source in Datable ​

  1. Navigate to the Sources page in Datable
  2. Select AWS CloudTrail from the available sources
  3. Provide the following configuration:
    • Source Name: A descriptive name
    • S3 Bucket: Your CloudTrail bucket name
    • AWS Region: The bucket's region
    • Access Key ID and Secret Access Key (or Role ARN)
    • SQS Queue URL (if using event notifications)
    • Authentication: Follow instructions to set up IAM user/role with necessary permissions or use existing credentials
  4. Click Save to create the source

Data Collected ​

CloudTrail logs contain:

Event Information ​

  • Event Time: When the API call was made
  • Event Name: The API action performed
  • Event Source: The AWS service
  • AWS Region: Where the event occurred

Identity Information ​

  • User Identity: Who made the API call
  • User Type: Root, IAM user, assumed role, or federated user
  • Source IP Address: Origin of the request
  • User Agent: Client application information

Request Details ​

  • Request Parameters: Input to the API call
  • Response Elements: API response data
  • Request ID: Unique identifier for the request

Additional Metadata ​

  • Error Code: If the API call failed
  • Error Message: Failure details
  • Resources: Affected AWS resources
  • Event Type: Management, Data, or Insights event

Configuration Options ​

Multi-Region Support ​

Configure collection from multiple regions by:

  • Setting up trails in each region
  • Configuring all trails to deliver to the same S3 bucket
  • Or creating separate sources for each regional bucket

Organization Trail ​

For AWS Organizations:

  • Enable organization trail for all accounts
  • Configure once at the organization level
  • Automatically collect logs from all member accounts

Event Filtering ​

Filter which events to process:

  • By event name patterns
  • By AWS service
  • By user identity
  • By error/success status

Troubleshooting ​

No Data Appearing ​

  • Verify CloudTrail is enabled and delivering logs
  • Check S3 bucket permissions
  • Ensure IAM credentials are valid
  • Verify bucket name and region are correct

Permission Errors ​

  • Review IAM policy has all required permissions
  • Check S3 bucket policy doesn't block access
  • Verify KMS permissions if bucket is encrypted

Delayed Data ​

  • Check CloudTrail delivery time (typically 5-15 minutes)
  • Verify S3 event notifications are configured
  • Review SQS queue for messages

Missing Events ​

  • Ensure CloudTrail is logging the desired event types
  • Check if events are filtered at the trail level
  • Verify multi-region setup if applicable

Security Best Practices ​

IAM Permissions ​

  • Use least privilege principle
  • Create dedicated IAM user/role for Datable
  • Avoid using root account credentials
  • Regularly rotate access keys

S3 Security ​

  • Enable S3 bucket encryption
  • Use bucket policies to restrict access
  • Enable S3 access logging
  • Configure MFA delete for production buckets

CloudTrail Security ​

  • Enable log file validation
  • Use KMS encryption for logs
  • Enable CloudTrail Insights
  • Configure log file integrity validation

Cost Considerations ​

  • CloudTrail: First trail is free, additional trails incur charges
  • S3 Storage: Standard S3 storage rates apply
  • S3 API Calls: GET requests for reading logs
  • Data Transfer: Potential egress charges depending on configuration

Performance Optimization ​

Batch Processing ​

  • Logs are processed in batches for efficiency
  • Typical processing latency: 1-2 minutes after S3 delivery

Compression ​

  • CloudTrail logs are gzip compressed
  • Datable handles decompression automatically

Support ​

For additional support with the AWS CloudTrail integration, please contact the Datable support team or refer to the AWS CloudTrail documentation.