Appearance

AWS VPC Flow Logs ​

Datable integrates with AWS VPC Flow Logs to collect and analyze network traffic data, providing visibility into network communications, security threats, and traffic patterns within your AWS Virtual Private Cloud.

How it works ​

The VPC Flow Logs integration ingests flow log data from S3 buckets or CloudWatch Logs, processing network traffic records to provide insights into your VPC network activity.

Prerequisites ​

  • VPC Flow Logs enabled on your VPCs, subnets, or network interfaces
  • S3 bucket or CloudWatch Log Group for flow log delivery
  • IAM permissions for accessing flow logs

Setup Instructions ​

Step 1: Enable VPC Flow Logs ​

  1. Navigate to VPC in the AWS Console
  2. Select your VPC, subnet, or network interface
  3. Go to the Flow logs tab
  4. Click Create flow log
  5. Configure:
    • Filter: ALL, ACCEPT, or REJECT traffic
    • Destination: S3 bucket
    • Log format: Default or custom format
    • Maximum aggregation interval: 1 minute (for real-time) or 10 minutes

Step 2: Configure S3 Destination ​

If using S3 as destination:

  1. Create or select an S3 bucket
  2. Configure bucket policy to allow VPC Flow Logs delivery:
json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AWSLogDeliveryWrite",
      "Effect": "Allow",
      "Principal": {
        "Service": "delivery.logs.amazonaws.com"
      },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::your-bucket/flow-logs/*"
    }
  ]
}

Step 3: Configure the Source in Datable ​

  1. Navigate to the Sources page in Datable
  2. Select AWS VPC Flow Logs from available sources
  3. Provide configuration:
    • Source Name: A descriptive name
    • S3 Bucket: Your flow logs bucket
    • S3 Prefix: Log file prefix (if configured)
    • AWS Region: Region of your VPC
    • Access Key ID and Secret Access Key
    • Log Format: Default or custom format string
    • Authentication: Follow instructions to set up IAM user/role with necessary permissions or use existing credentials
  4. Click Save to create the source

Data Collected ​

Default Flow Log Fields ​

  • version: Flow log version
  • account-id: AWS account ID
  • interface-id: Network interface ID
  • srcaddr: Source IP address
  • dstaddr: Destination IP address
  • srcport: Source port
  • dstport: Destination port
  • protocol: IANA protocol number
  • packets: Number of packets
  • bytes: Number of bytes
  • start: Start time of flow
  • end: End time of flow
  • action: ACCEPT or REJECT
  • log-status: OK, NODATA, or SKIPDATA

Additional Available Fields ​

  • vpc-id: VPC identifier
  • subnet-id: Subnet identifier
  • instance-id: EC2 instance ID
  • tcp-flags: TCP flags
  • type: IPv4, IPv6, or EFA
  • pkt-srcaddr: Packet source address
  • pkt-dstaddr: Packet destination address
  • region: AWS region
  • az-id: Availability zone ID
  • flow-direction: Ingress or egress
  • traffic-path: Network path taken

Configuration Options ​

Custom Log Format ​

Define custom formats to include additional fields:

${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status} ${vpc-id} ${subnet-id} ${instance-id} ${tcp-flags} ${type} ${pkt-srcaddr} ${pkt-dstaddr}

Traffic Filtering ​

Configure flow logs to capture:

  • ALL: All traffic
  • ACCEPT: Only accepted traffic
  • REJECT: Only rejected traffic

Aggregation Interval ​

  • 1 minute: Near real-time visibility (higher cost)
  • 10 minutes: Standard aggregation (lower cost)

Use Cases ​

Security Monitoring ​

  • Detect unauthorized access attempts
  • Identify suspicious traffic patterns
  • Monitor security group effectiveness
  • Track lateral movement

Network Troubleshooting ​

  • Diagnose connectivity issues
  • Analyze packet loss
  • Identify bottlenecks
  • Verify routing behavior

Compliance & Auditing ​

  • Maintain network audit trails
  • Demonstrate compliance controls
  • Generate traffic reports
  • Monitor data exfiltration

Cost Optimization ​

  • Analyze data transfer costs
  • Identify chatty applications
  • Optimize NAT gateway usage
  • Monitor cross-AZ traffic

Troubleshooting ​

No Flow Logs Appearing ​

  • Verify Flow Logs are enabled and active
  • Check S3 bucket permissions
  • Ensure IAM credentials are valid
  • Confirm traffic is occurring in the VPC

Missing Traffic ​

  • Check flow log filter (ALL vs ACCEPT/REJECT)
  • Verify all interfaces have flow logs enabled
  • Review aggregation interval settings
  • Ensure log-status is not SKIPDATA

Permission Errors ​

  • Review IAM policy permissions
  • Check S3 bucket policy
  • Verify KMS permissions if encrypted

High Volume Issues ​

  • Consider using REJECT filter for security monitoring
  • Increase aggregation interval
  • Implement S3 lifecycle policies
  • Use partitioned S3 prefixes

Performance Considerations ​

Data Volume ​

VPC Flow Logs can generate significant data:

  • Estimate: 1-10 MB per ENI per hour
  • Use filters to reduce volume
  • Consider sampling for high-traffic environments

Processing Latency ​

  • Flow logs delivery: 5-15 minutes
  • Processing time: 1-2 minutes after delivery
  • Use 1-minute intervals for near real-time

Cost Optimization ​

Flow Logs Costs ​

  • Data ingestion: Per GB ingested
  • S3 storage: Standard S3 rates
  • Data transfer: Potential egress charges

Optimization Strategies ​

  • Use appropriate aggregation intervals
  • Filter unnecessary traffic
  • Implement S3 lifecycle policies
  • Archive old logs to Glacier
  • Use S3 Intelligent-Tiering

Security Best Practices ​

Access Control ​

  • Use least privilege IAM policies
  • Enable MFA for administrative access
  • Rotate access keys regularly
  • Use IAM roles over users

Data Protection ​

  • Encrypt flow logs at rest
  • Use S3 bucket encryption
  • Enable S3 access logging
  • Implement bucket versioning

Monitoring ​

  • Set up CloudWatch alarms
  • Monitor unusual traffic patterns
  • Track configuration changes
  • Review access logs regularly

Advanced Features ​

Enhanced VPC Flow Logs ​

Consider enabling enhanced fields for:

  • Traffic path analysis
  • NAT gateway optimization
  • Transit gateway monitoring
  • VPC endpoint usage

Integration with Other Services ​

  • Combine with CloudTrail for complete visibility
  • Correlate with GuardDuty findings
  • Integrate with Security Hub
  • Use with Network Insights

Support ​

For additional support with the AWS VPC Flow Logs integration, please contact the Datable support team or refer to the AWS VPC Flow Logs documentation.