S3 Integration Guide

This guide will show you how to integrate Amazon S3 with the Secure60 platform. This integration enables you to collect and analyze logs and events stored in S3 buckets, making them available for security analysis and monitoring through Secure60.

Overview

This guide will explain how to configure the Secure60 Collector to pull data from S3 buckets using SQS notifications. The integration automatically normalizes S3 event data into the Secure60 Common Information Model (CIM) format, making it easy to analyze and correlate with other security data sources.

Prerequisites

1. An AWS account with S3 and SQS access (or S3-compatible storage provider)
2. An S3 bucket containing the logs you want to collect
3. An SQS queue configured to receive S3 bucket notifications
4. A Secure60 Collector instance (see Secure60 Collector documentation)
5. AWS credentials with appropriate permissions

Configuration Steps

1. Set Up S3 to SQS Notifications

   • Create an SQS queue in your AWS account
   • Configure S3 bucket notifications to send events to the SQS queue
   • Ensure the SQS queue has the necessary permissions to receive S3 notifications

2. Configure Secure60 Collector

   Create a custom transformation file named transform-s3.yaml with the following content:

   sources:
     source_s3:
       type: aws_s3
       region: ${AWS_REGION:?err}
       auth:
         access_key_id: ${AWS_ACCESS_KEY_ID:?err}
         secret_access_key: ${AWS_SECRET_ACCESS_KEY:?err}
       sqs:
         queue_url: ${SQS_QUEUE_URL:?err}
         delete_message: true
         visibility_timeout_secs: 300
       compression: auto
       decoding:
         codec: json
         lossy: true
   
   transforms:
     transform_s3:
       inputs:
         - source_s3
       type: remap
       source: |
         .vendor = "aws"
         .product = "S3"
         .type = "cloud"
         .app_name = "s3"
         .technology_group = "cloud"
   
         # Extract bucket and object information
         .source_name = .bucket
         .source_type = .object
   
         # Parse timestamp if available
         if exists(.timestamp) {
           .event_time = .timestamp
         }
   
         # Extract additional metadata
         if exists(.message) {
           .message_text = .message
         }
   
         # Normalize region information
         if exists(.region) {
           .source_location = .region
         }
   
         # Clean up original fields
         del(.bucket)
         del(.object)
         del(.region)
         del(.source_type)
   
         # Compact the event
         compact(., recursive:true, null:true, string:true, object:true, array:true, nullish:true)      
   

3. Configure Environment Variables

   Add the following environment variables to your Secure60 Collector .env file:

   AWS_ACCESS_KEY_ID=your_access_key
   AWS_SECRET_ACCESS_KEY=your_secret_key
   AWS_REGION=your_region
   SQS_QUEUE_URL=your_sqs_queue_url
   

4. Deploy the Collector

   Mount the custom transformation file and update your Docker run command:

   docker run -i --name s60-collector \
     -v ./transform-s3.yaml:/etc/vector/transforms-active/transform-s3.yaml \
     -p 80:80 -p 443:443 -p 514:514 -p 6514:6514 \
     --rm -d --env-file .env secure60/s60-collector:1.07
   

   Or using Docker Compose:

   services:
     s60-collector:
       image: "secure60/s60-collector:1.07"
       volumes:
         - ./transform-s3.yaml:/etc/vector/transforms-active/transform-s3.yaml
       ports:
         - "443:443"
         - "80:80"
         - "514:514"
         - "6514:6514"
       env_file:
         - .env
   

Data Normalization

The custom transformation automatically normalizes S3 events into the Secure60 Common Information Model. This includes:

• Event timestamps
• Bucket and object information
• Region and location data
• Message content
• Source and destination details

Verification

To verify the integration is working:

1. Check the Secure60 Collector logs:

   docker logs s60-collector
   

2. View events in the Secure60 Portal:

   • Navigate to the Search page
   • Filter for events with vendor="aws" and product="S3"

Troubleshooting

Common issues and solutions:

1. Authentication Errors

   • Verify AWS credentials are correct
   • Ensure the IAM user/role has necessary permissions
   • Check if credentials are properly mounted in the container

2. SQS Connection Issues

   • Verify the SQS queue URL is correct
   • Ensure the queue is properly configured to receive S3 notifications
   • Check network connectivity to AWS services

3. Data Not Appearing

   • Verify S3 bucket notifications are properly configured
   • Check SQS queue for messages
   • Review Secure60 Collector logs for transformation errors

For additional assistance, contact Secure60 Support at support@secure60.io