Create an SQS Queue and Configure S3 ObjectCreated Notifications

Finding the S3 Bucket that Contains the Data that You Want to Collect

You must find the S3 bucket that contains the data that you want to collect.

1. Log in to the AWS Management Console as an administrator.

2. Click Services, and then navigate to the Simple Queue Service Management Console.

3. From the Region column in the S3 buckets list, note the region where the bucket that you want to collect data from is located.

4. Enable the check box beside the bucket name, and then from the panel that opens to the right, click Copy Bucket ARN to copy the value to the clipboard. Save this value or leave it on the clipboard. You will need this value when you set up SQS queue permissions.

Creating the SQS Queue that is used to Receive ObjectCreated Notifications

You must create an SQS queue and configure S3 ObjectCreated notifications in the AWS Management Console when using the Amazon AWS REST API protocol.

You must complete Finding the S3 Bucket that Contains the Data that You Want to Collect. The SQS Queue must be in the same region as the AWS S3 bucket that the queue is collecting from.

1. Log in to the AWS Management Console as an administrator.

2. Click Services, and then navigate to the Simple Queue Service Management Console.

3. In the upper right of the window, change the region to where the bucket is located. You noted this value when you completed the Finding the S3 Bucket that Contains the Data that You Want to Collect procedure.

4. Select Create New Queue, and then type a value for the Queue Name.

5. Click Standard Queue, and then select Configure Queue at the bottom of the window. Change the default values for the following Queue Attributes.

   • Default Visibility Timeout - 60 seconds (Lower can be used. However, in the case of load balanced collection, duplicate events might occur with values of less than 30 seconds. This value can't be 0.)

   • Message Retention Period - 14 days (Lower can be used. However, in the event of an extended collection, data might be lost.)

   Use the default value for the remaining Queue Attributes.

   More options such as Redrive Policy or SSE can be used depending on the requirements for your AWS environment. These values should not affect collection of data.

6. Select Create Queue.

Setting up SQS Queue Permissions

You must set up SQS queue permissions for users to access the queue.

You must complete Creating the SQS Queue that is used to Receive ObjectCreated Notifications.

1. Log in to the AWS Management Console as an administrator.

2. Go to the SQS Management Console, and then select the queue that you created from the list.

3. From the Properties window, select Details. Record the ARN field value.

   Example: arn:aws:sqs:us-east-1:123456789012:MySQSQueueName

4. Set the SQS queue permissions by using either the Permissions Editor or a JSON policy document.

   • Using the Permissions Editor:

     1. From the Properties window, select Permissions > Add a Permission, and then configure the following options.

        Table 1: Permission Parameters

        Principal	Click Everybody (*)
        Actions	From the list, select SendMessage
        Effect	Click Allow

     2. Click Add Conditionals (Optional), and then configure the following parameters:

        Table 2: Add Conditionals (Optional) Parameters

        Qualifier	None
        Condition	ARNLike
        Key	aws:SourceArn
        Value	ARN of the S3 bucket, from Finding the S3 Bucket that Contains the Data that You Want to Collect Example: aws:s3:::my-examples3bucket

     3. Click Add Condition.

     4. Click Add Permission.

   • Using a JSON Policy Document:

     1. In the Properties window, at the bottom, select Edit Policy Document (Advanced).

     2. Copy and paste the following JSON policy into the Edit Policy Document window:

        Copy and paste might not preserve the whitespace in the JSON policy. The whitespace is required. If the whitespace is not preserved when you paste the JSON policy, paste it into a text editor and restore the whitespace. Then, copy and paste the JSON policy from your text editor into the Edit Policy Document window.

     3. Change the Resource in this policy document to match the ARN of your SQS queue from 3, and the "aws:SourceArn" to match the ARN of your bucket that you recorded when you completed the Finding the S3 Bucket that Contains the Data that You Want to Collect procedure.

5. Click Review Policy. Ensure the data is correct, and then click Save Changes.

Creating ObjectCreated Notifications

You must create ObjectCreated notifications for the folders that you want to monitor in the bucket.

1. Log in to the AWS Management Console as an administrator.

2. Click Services, then navigate to the Simple Queue Service Management Console.

3. Select a bucket.

4. Click the Properties tab.

5. In the Events pane, click Add notification and then configure the parameters for the new event.

   The following table shows an example of an ObjectCreated notification parameter configuration:

   Table 3: Example: New ObjectCreated Notification Parameter Configuration

   Parameter	Value
   Name	Type a name of your choosing.
   Events	Select All object create events.
   Prefix	AWSLogs/ Tip: You can choose a prefix that contains the data that you want to find, depending on where the data is located and what data that you want to go to the queue. For example, AWSLogs/, CustomPrefix/AWSLogs/, AWSLogs/ 123456789012/.
   Suffix	json.gz
   Send to	SQS queue Tip: You can send the data from different folders to the same or different queues to suit your collection or JSA tenant needs. Choose one or more of the following methods: Different folders that go to different queues Different folders from different buckets that go to the same queue Everything from a single bucket that goes to a single queue Everything from multiple buckets that go to a single queue
   SQS	The Queue Name from 4of Creating the SQS Queue that is used to Receive ObjectCreated Notifications.

   In the example in figure 1 of a parameter configuration, notifications are created for AWSLogs/ from the root of the bucket. When you use this configuration, All ObjectCreated events trigger a notification. If there are multiple accounts and regions in the bucket, everything gets processed. In this example, json.gz is used. This file type can change depending on the data that you are collecting. Depending on the content in your bucket, you can omit the extension or choose an extension that matches the data you are looking for in the folders where you have events set up.

   After approximately 5 minutes, the queue that contains data displays. In the Messages Available column, you can view the number of messages.

6. Click Services, then navigate to Simple Queue Services.

7. Right-click the Queue Name from 4 of Creating the SQS Queue that is used to Receive ObjectCreated Notifications, then select View/Delete Messages to view the messages.

   Sample message:

8. Click Services, then navigate to IAM.

9. Set a User or Role permission to access the SQS queue and for permission to download from the target bucket. The user or user role must have permission to read and delete from the SQS queue. For information about adding, managing and changing permissions for IAM users, see the IAM Users documentation. After JSA reads the notification and then downloads and processes the target file, the message must be deleted from the queue.

   Sample Policy:

   You can add multiple buckets. To ensure that all objects are accessed, you must have a trailing /* at the end of the folder path that you added.

   You can add this policy directly to a user, a user role, or you can create a minimal access user with sts:AssumeRole only. When you configure a log source in JSA, configure the assume Role ARN parameter for JSA to assume the role. To ensure that all files waiting to be processed in a single run (emptying the queue) can finish without retries, use the default value of 1 hour for the API Session Duration parameter.

   When using assumed roles, ensure that the ARN of the user assuming the rule is in the Trusted Entities for that role. From the Trusted entities pane, you can view the trusted entities that can assume the role. In addition, the user must have permission to assume roles in that (or any) account.