How To Sync AWS S3 Buckets Using RClone

Author: erics, September 9th, 2021

The aws s3 sync command is slow and painful! I needed a more efficient way to sync to large buckets (prod to dev). Finally settled on RClone: https://rclone.org/docs/

cd {extracted zip dir}
./rclone config 
cat ~/.config/rclone/rclone.conf

1

2

3

cd {extracted zip dir}

./rclone config

cat ~/.config/rclone/rclone.conf

During rclone config I called remote “s3” ;-}

./rclone sync -v --progress --fast-list --checksum s3:wyzaerd-demo-prod s3:wyzaerd-demo-dev

1	./rclone sync -v --progress --fast-list --checksum s3:wyzaerd-demo-prod s3:wyzaerd-demo-dev

Categories: How-To's, Technology Tags: AWS, Bucket, howto, rclone, rsync, S3, s3s3, s3s3mirror, Sync, tips | No comments

How To Allow IAM Users AWS CDN Read/Write Access For W3 Total Cache

Author: erics, July 28th, 2021

SUMMARY: Needed to create an AWS IAM Policy to allow a user access to two buckets and their associated CDN’s in WordPress W3 Total Cache

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::wyzaerd-demo-prod"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::wyzaerd-demo-prod/*"
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::wyzaerd-demo-dev"
        },
        {
            "Sid": "VisualEditor3",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::wyzaerd-demo-dev/*"
        },
        {
            "Sid": "VisualEditor4",
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "cloudfront:ListDistributions"
            ],
            "Resource": "*"
        }
    ]
}

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

{

"Version": "2012-10-17",

"Statement": [

{

"Sid": "VisualEditor0",

"Effect": "Allow",

"Action": "s3:ListBucket",

"Resource": "arn:aws:s3:::wyzaerd-demo-prod"

},

{

"Sid": "VisualEditor1",

"Effect": "Allow",

"Action": [

"s3:PutObject",

"s3:GetObject",

"s3:DeleteObject",

"s3:PutObjectAcl"

],

"Resource": "arn:aws:s3:::wyzaerd-demo-prod/*"

},

{

"Sid": "VisualEditor2",

"Effect": "Allow",

"Action": "s3:ListBucket",

"Resource": "arn:aws:s3:::wyzaerd-demo-dev"

},

{

"Sid": "VisualEditor3",

"Effect": "Allow",

"Action": [

"s3:PutObject",

"s3:GetObject",

"s3:DeleteObject",

"s3:PutObjectAcl"

],

"Resource": "arn:aws:s3:::wyzaerd-demo-dev/*"

},

{

"Sid": "VisualEditor4",

"Effect": "Allow",

"Action": [

"s3:ListAllMyBuckets",

"cloudfront:ListDistributions"

],

"Resource": "*"

}

]

}

Categories: How-To's, Technology Tags: Amazon, AWS, Bucket, CDN, CloudFront, howto, IAM, JSON, Policy, S3, tips, User, W3 Total Cache | No comments

How To Sync Box to S3 Using rclone

Author: erics, October 3rd, 2019

To sync various cloud resources, use the excellent cli tool rclone from https://rclone.org/docs/ For this use case, the need was to sync from Box to an AWS S3 bucket. Install rclone:

curl https://rclone.org/install.sh | sudo bash

1	curl https://rclone.org/install.sh \| sudo bash

Configure both S3 and Box – for remote name labels I just used “S3” and “Box”:

rclone config

1	rclone config

Validate Access and Functionality:

rclone lsd Box:
rclone lsd S3:

1 2	rclone lsd Box: rclone lsd S3:

Perform […]

Categories: How-To's, Technology Tags: Amazon, AWS, AWS S3, Box, Clone, rclone, rsync, S3, Sync | No comments

How To install Boto on Amazon Linux using yum and pip

Author: erics, August 15th, 2013

yum install python
python /usr/bin/easy_install pip
pip install boto

1

2

3

yum install python

python /usr/bin/easy_install pip

pip install boto

Categories: How-To's, Technology Tags: Amazon, AWS, Boto, EC2, howto, Install, Linux, pip, Python, S3, tips, Yum | No comments

How To Install s3cmd

Author: erics, June 7th, 2013

Install Procedure Download and extract into /usr/local: http://downloads.sourceforge.net/project/s3tools/s3cmd/1.0.1/s3cmd-1.0.1.tar.gz

cd /usr/local/bin
ln -s /usr/local/s3cmd-1.0.1/s3cmd
/usr/local/bin/s3cmd --configure
Access Key: {ACCESS_KEY}
Secret Key: {SECRET_KEY}
Encryption password: {PASSWORD}

1

2

3

4

5

6

cd /usr/local/bin

ln -s /usr/local/s3cmd-1.0.1/s3cmd

/usr/local/bin/s3cmd --configure

Access Key: {ACCESS_KEY}

Secret Key: {SECRET_KEY}

Encryption password: {PASSWORD}

Configuration saved to ‘/root/.s3cfg’ Command Line Examples List all buckets: s3cmd ls Make a new bucket in the default region (US): s3cmd mb s3://{YOUR_BUCKET_HERE} Make a new bucket in a specific region: s3cmd mb –bucket-location={DATACENTER} s3://{YOUR_BUCKET_HERE} As of now the datacenters are: US (default), EU, us-west-1, […]

Categories: How-To's, Technology Tags: Amazon, AWS, howto, Install, rsync, S3, s3cfg, s3cmd, s3sync, s3tools, Sync, tips | No comments

How To Copy An Entire Amazon S3 Bucket

Author: erics, May 31st, 2012

1. You need Ruby, rubygems and the right_aws gem. yum install ruby rubygems gem update –system gem install right_aws 2. Replace any values inside curly braces {} below with real values and remove the curly braces. YMMV

require 'rubygems'
require 'right_aws'

S3ID = "{S3 ID GOES HERE}"
S3KEY = "{S3 KEY GOES HERE}"
SRCBUCKET = "{your-s3-bucket-1}"
DESTBUCKET = "{your-s3-bucket-2}"

s3 = RightAws::S3Interface.new(S3ID, S3KEY)
s3.incrementally_list_bucket(SRCBUCKET) do |h|
h[:contents].each do |o|
puts('Copying ' + o[:key])
s3.copy(SRCBUCKET, o[:key], DESTBUCKET, o[:key])
end
end
puts('Done.')

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

require 'rubygems'

require 'right_aws'

S3ID = "{S3 ID GOES HERE}"

S3KEY = "{S3 KEY GOES HERE}"

SRCBUCKET = "{your-s3-bucket-1}"

DESTBUCKET = "{your-s3-bucket-2}"

s3 = RightAws::S3Interface.new(S3ID, S3KEY)

s3.incrementally_list_bucket(SRCBUCKET) do |h|

h[:contents].each do |o|

puts('Copying ' + o[:key])

s3.copy(SRCBUCKET, o[:key], DESTBUCKET, o[:key])

end

puts('Done.')

Categories: How-To's, Technology Tags: Amazon, AWS, Bucket, Copy, gem, right_aws, Ruby, rubygems, S3 | No comments

How To Cache WordPress Using Amazon S3 & CloudFront CDN

Author: erics, May 3rd, 2012

I found a fantastic tutorial on using the W3 Total Cache plugin with Amazon’s S3/CDN that I had to share: http://www.makeuseof.com/tag/give-wordpress-blog-lightning-fast-speeds-w3-total-cache/ The basic gist of the article says to do the following: Install and activate the W3 Total Cache plugin Sign up for Amazon Web Services (AWS) Obtain your AWS Access and Secret key pair […]

Categories: How-To's, Technology Tags: Amazon, AWS, Cache, CDN, CloudFront, howto, S3, tips, WordPress | No comments

How To Install s3fs On Amazon Linux Using yum

Author: erics, April 2nd, 2012

To be performed as the root user:

yum install gcc libstdc++-devel gcc-c++ curl-devel libxml2-devel openssl-devel mailcap fuse fuse-devel
wget http://s3fs.googlecode.com/files/s3fs-1.61.tar.gz
tar xvzf s3fs-1.61.tar.gz
cd s3fs-1.61/
./configure --prefix=/usr
make
make install
vim  /etc/passwd-s3fs
chmod 640 /etc/passwd-s3fs
s3fs yourBucketNameHere /mnt

1

2

3

4

5

6

7

8

9

10

yum install gcc libstdc++-devel gcc-c++ curl-devel libxml2-devel openssl-devel mailcap fuse fuse-devel

wget http://s3fs.googlecode.com/files/s3fs-1.61.tar.gz

tar xvzf s3fs-1.61.tar.gz

cd s3fs-1.61/

./configure --prefix=/usr

make

make install

vim /etc/passwd-s3fs

chmod 640 /etc/passwd-s3fs

s3fs yourBucketNameHere /mnt

http://code.google.com/p/s3fs/wiki/FuseOverAmazon The s3fs password file has this format (use this format if you have only one set of credentials): accessKeyId:secretAccessKey If have more than one set of credentials, then you can have default credentials as specified above, but this syntax will be recognized as well: bucketName:accessKeyId:secretAccessKey

Categories: How-To's, Technology Tags: Amazon, Amazon Linux, AWS, fuse, gcc, howto, S3, s3fs, tips, Yum | 5 comments

How To Sync AWS S3 Buckets Using RClone

How To Allow IAM Users AWS CDN Read/Write Access For W3 Total Cache

How To Sync Box to S3 Using rclone

How To install Boto on Amazon Linux using yum and pip

How To Install s3cmd

How To Copy An Entire Amazon S3 Bucket

How To Cache WordPress Using Amazon S3 & CloudFront CDN

How To Install s3fs On Amazon Linux Using yum

Our Link Network

Business

Client Websites

Fine Dining

Friends & Family

Galleries

Health

Resources

Technology

Travel

The Archives

Latest Ramblings…

Various Topics of No Interest