yuta/llm-wiki

Fork 0

Files

T

yuta 87eacd39b2 add

2026-06-30 22:22:00 +09:00

18 KiB

Raw Blame History

source_url, ingested, sha256, discovered_from

source_url

ingested

sha256

discovered_from

https://d1.awsstatic.com/events/Summits/reinvent2023/STG314_Dive-deep-on-Amazon-S3.pdf

2026-06-30

970f9400ef22f30b45946dcacdeea413b8061d2553aafd5d2c52e1067c2d3dda

platform	channel_id	channel_name	message_id	author_id	posted_at	message_excerpt
discord	1028287639918497822	chat	1521490006915678228	890908900520505354	2026-06-30T12:18:05.566000000Z	https://d1.awsstatic.com/events/Summits/reinvent2023/STG314_Dive-deep-on-Amazon-S3.pdf

STG314

Dive deep on Amazon S3

Amy Therrien

Seth Markle

(she/her) Director S3 Engineering Amazon S3

(he/him) Senior Principal Engineer Amazon S3

There is no compression algorithm for experience

Three epochs

Reactive

Threat modeling

Proactive

Culture: Threat reviews

•

Actual written document

•

Interactive working meeting

•

Review with experienced team members

•

Learning opportunity for junior engineers

350+ microservices All AWS Regions

Clients

Front end Web servers, DNS, and network

Index Key/value mapping to storage

The S3 front end is big

The S3 front end is big Traffic peaks at over 1PB/sec

The threat of scale and going wide Mechanisms for mitigation

Use multipart uploads to parallelize puts

Use ranges to parallelize gets

Spread requests across many IPs in the fleet

MPU and ranges 100 MB

100 MB

X MB/s

Break into parts and parallelize data transfer in 100 MB 20 MB 20 MB 20 MB

20 MB 20 MB

100 MB X MB/s

X MB/s

Break into parts and parallelize data transfer USING

THE

MULTIPART

UPLOAD

API

create-multipart-upload [--acl ] [--grant-full-control ] [--grant-read ] [--grant-read-acp ] [--grant-write-acp ] [--metadata ] [--storage-class ] [--server-side-encryption ] [--ssekms-key-id ] [--bucket-key-enabled] [--tagging ] [--object-lock-mode ]

upload-part --part-number --upload-id --key --bucket

n times

upload-part --part-number --upload-id --key --bucket

complete-multipart-upload --checksum-algorithm --key --bucket

--upload-id --key --bucket

S3API CLI docs:

Same thing in reverse for data transfer out 100 MB

100 MB X MB/s

X MB/s

20 MB 20 MB

20 MB 20 MB 20 MB

Same thing in reverse for data transfer out USE

GET-OBJECT-ATTRIBUTES

HEAD-OBJECT

DERIVE

RANGES

get-object-attributes … "Parts": [ { "PartNumber": 1, "Size": 17179870, "ChecksumCRC32": "fay0Cw==" }, {

"PartNumber": 2, "Size": 17179870, "ChecksumCRC32": "4qs8xw==" }, {

Calculate offsets or just use part numbers

"PartNumber": 3, "Size": 14692009, "ChecksumCRC32": "Ajz83g==" }

S3API CLI docs:

Same thing in reverse for data transfer out USING

RANGE

GET

REQUESTS

get-object [--range ] [--part-number ]

n times

--key --bucket

… stitch back together and write to storage in code Range GET RFC:

S3API CLI docs:

Spreading requests across the fleet IT’S

ALWAYS

DNS

Spreading requests across the fleet IT’S

ALWAYS

DNS

nslookup s3.amazonaws.com

Server: 192.0.2.3 Address: 192.0.2.3#53

PUT

Spreading requests across the fleet IT’S

ALWAYS

DNS

nslookup s3.amazonaws.com

Server: 192.0.2.3 Address: 192.0.2.3#53

Spreading requests across the fleet IT’S

ALWAYS

DNS

nslookup s3.amazonaws.com

Server: 192.0.2.3 Address: 192.0.2.3#53

PUT

Non-authoritative answer: Name: s3.amazonaws.com Address: 192.0.2.1 Name: s3.amazonaws.com Address: 192.0.2.4 Name: s3.amazonaws.com Address: 192.0.2.5 Name: s3.amazonaws.com Address: 192.0.2.9

Name: s3.amazonaws.com Address: 192.0.2.2 Name: s3.amazonaws.com Address: 192.0.2.6 Name: s3.amazonaws.com Address: 192.0.2.8 Name: s3.amazonaws.com Address: 192.0.2.7

Multi-value answers NEW

Common Runtime (CRT) BEST

PRACTICES

IMPLEMENTED

CODE

Open source component in the AWS SDK • Automated multipart uploads

• Parallelization of range GETs for downloads • Built-in retry logic using multiple IPs

• And more

S3AsyncClient.crtCreate();

CRT info:

NEW

Mountpoint

Mountpoint with Local Cache

NEW

Mountpoint for Containers

The Mountpoint for Amazon S3 Family

NEW

Mountpoint for S3 Express

Mountpoint in action CMP214

HPC on AWS for semiconductors and healthcare life sciences With access to scalable, cost-effective supercomputing capabilities in the cloud, organizations can accelerate discoveries in semiconductors, biopharma, and medicine

Clients

Front end Web servers, DNS, and network

Index Key/value mapping to storage

The S3 index is big

The S3 index is big 350 trillion objects 100+ million requests per second

Usage

Capacity

A-F

G-M

N-S

T-Z

A-F

G-M

Q-S

N-P SPLIT

T-Z

reinvent-bucket

What’s a prefix?

reinvent-bucket

What’s a prefix? Any string of characters after the bucket name

What’s a prefix? reinvent-bucket/p reinvent-bucket/prefix reinvent-bucket/prefix1/ reinvent-bucket/prefix2/ reinvent-bucket/prefix1/data/other reinvent-bucket/prefix1/data/otherstuff © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.

3,500 PUT requests 5,500 GET requests per prefix

reinvent-bucket/prefix1/a

reinvent-bucket/prefix1 5,500 GETs/second

reinvent-bucket 5,500 GETs/second

5,500 GETs/second SPLIT

reinventbucket/prefix1/b 5,500 GETs/second

SPLIT

reinvent-bucket/prefix2/a

reinvent-bucket/prefix2 5,500 GETs/second

5,500 GETs/second SPLIT

reinventbucket/prefix2/b 5,500 GETs/second

Usage

Capacity

Maximizing TPS with good key naming Mechanisms for mitigation

Keep cardinality to the left in key names

Keep dates to the right in key names

Key naming example (switch around day/prefix) reinvent-bucket/day1/prefix1/a 5,500 GETs/second

reinvent-bucket/day1/prefix1

SPLIT

reinvent-bucket/day1/prefix1/b

5,500 GETs/second

reinvent-bucket/day1 5,500 GETs/second

5,500 GETs/second SPLIT

reinvent-bucket/day1/prefix2/a

reinvent-bucket/day1/prefix2 5,500 GETs/second

5,500 GETs/second SPLIT

reinvent-bucket/day1/prefix2/b 5,500 GETs/second

Date far to the left in key name!!

Key naming example reinvent-bucket/day1/prefix1/a 5,500 GETs/second

reinvent-bucket/day1/prefix1

SPLIT

reinvent-bucket/day1/prefix1/b

5,500 GETs/second

reinvent-bucket/day1 5,500 GETs/second

5,500 GETs/second SPLIT

reinvent-bucket/day1/prefix2/a

reinvent-bucket/day1/prefix2 5,500 GETs/second

5,500 GETs/second SPLIT

reinvent-bucket/day1/prefix2/b 5,500 GETs/second

reinvent-bucket/day2 5,500 GETs/second

• The partitions from day1 are now unused • We will need to split day2 as it sees sustained load • Likely throttling as that process occurs

Key naming example Date far to the right in key name!! reinvent-bucket/prefix1/a/day1 reinvent-bucket/prefix1/a/day2 SPLIT

reinvent-bucket/prefix1

reinvent-bucket/prefix1/b/day1 reinvent-bucket/prefix1/b/day2

5,500 GETs/second

reinvent-bucket/prefix 5,500 GETs/second

5,500 GETs/second

5,500 GETs/second SPLIT

reinvent-bucket/prefix2/a/day1 reinvent-bucket/prefix2/a/day2

reinvent-bucket/prefix2 5,500 GETs/second

SPLIT

5,500 GETs/second

reinvent-bucket/prefix2/b/day1 reinvent-bucket/prefix2/b/day2 5,500 GETs/second

Clients

Front end Web servers, DNS, and network

Index Key/value mapping to storage

We store a lot of data

We store a lot of data Millions of hard drives Exabytes of data

99.999999999% DATA DURABILITY

Threat: Device failure

We must protect data stored on drives that can fail or corrupt bits at rest

Achieving 11 9s of durability

End-to-end integrity checking of requests

Data always stored on redundant devices

Periodic durability auditing for data at rest

End-to-end integrity checking of requests Data redundantly stored and checksums taken in storage Checksums taken in transit

device device

S3:PutObject 200:SUCCESS

device device

Data stored compared to data uploaded

Data always stored on redundant devices 5 devices, each with different data

A X B D

A C D

A C E

B C E

B D E

Data always stored on redundant devices 5 devices, each with different data

A X B D

A B D

A C D

A C E

B C E

B D E

Data always stored on redundant devices

Now do it at scale • Adequate spare capacity always on hand • Aggressive monitoring for failure • Redundancy maintained when hardware inevitably fails

Periodic durability auditing for data at rest

11 9s of durability

λt

-2 µt βt(-2)

λt 0

-1 µt βt(-1)

µt βt(0)

Λtσt(C)

λt

1 µt

C µt

Λtσt(C+1) Λtσt(C+2) C+2

C+1

µt

Threat model: AZ loss

We must protect data we store against the unexpected total or partial loss of a zone. We assume that any single facility may fail at any time and that we must protect the durability of data that is stored within it.

Multi-AZ by default Multi-AZ by default AZ1

AZ3

AZ2

Multi-AZ by default

200:SUCCESS

AZ1

AZ2

AZ3

Amazon S3 Express One Zone

Full or partial loss of an Availability Zone may lose my data in S3 Express One Zone

99.999999999% data durability WITHIN

SINGLE

AVAILABILITY

ZONE

AZ1

AZ2

AZ3

99.999999999% data durability WITHIN

SINGLE

AVAILABILITY

ZONE

AZ2

99.999999999% data durability WITHIN

SINGLE

AVAILABILITY

ZONE

End-to-end integrity checking of requests

Data always stored on redundant devices

Periodic durability auditing for data at rest

Loss or damage to all or part of an AZ

Accidental delete WHEN

YOU

TELL

DELETE

SOMETHING,

We never want accidental deletion of data in Amazon S3 buckets, especially due to bulk deletion operations

Accidental delete

Mechanisms for mitigation

S3 Versioning

S3 Replication

S3 Object Lock

Backups

We are a global service

We are a global service Operating in all AWS Regions Regional storage classes span 3+ AZs

The power to touch is the power to destroy

The availability of one AWS Region can never affect the availability of another

Regional isolation A

LEARNED

TENET

Amazon S3: 2006 - 2010

AZ failure threat model

We must remain available through the unexpected loss of an entire zone. We assume that any single facility may fail at any time and that we must continue to serve requests when an entire zone is offline.

Availability during zone failure

200:SUCCESS

AZ1

AZ2

AZ3

Availability during zone failure IT’S

ALWAYS

DNS

nslookup s3.amazonaws.com

Server: 192.0.2.3 Address: 192.0.2.3 #53 Non-authoritative answer:

PUT

Name: s3.amazonaws.com Address: 192.0.2.4

Name: s3.amazonaws.com Address: 192.0.2.5

Name: s3.amazonaws.com Address: 192.0.2.6 Name: s3.amazonaws.com Address: 192.0.2.7 Name: s3.amazonaws.com Address: 192.0.2.8 Name: s3.amazonaws.com Address: 192.0.2.117 Name: s3.amazonaws.com Address: 192.0.2.232 Name: s3.amazonaws.com Address: 192.0.2.64

AZ fault tolerance: Not just for AZ faults!

• Software deployments • New hardware adoption • Configuration changes • More Separate fault domains

Defense in depth Building good guardrails • Build for correctness • Threats and mitigations • But assume incorrectness • Guardrails

Examples

To recap

No compression algorithm for experience

Thank you! Amy Therrien

Seth Markle

[email protected]

Please complete the session survey in the mobile app

18 KiB Raw Blame History Unescape Escape

nslookup s3.amazonaws.com

nslookup s3.amazonaws.com

nslookup s3.amazonaws.com

nslookup s3.amazonaws.com

18 KiB

Raw Blame History