something very similar in mainframe land where they call files “datasets.” if you call a dataset a file, get ready to get an earfull!
which is ironic considering IBM themselves often call them files and one of the most popular dataset utilities is called FileAid. but if we acknowledge that, we lose a valuable opportunity to belittle and exclude newcomers
You know, the same deal kind of applies about “losing newcomers”. Because I avooded using AWS because everything seemed cryptic and weird with its terminology. So other companies got my business.
There is a reason for that since S3 is not block storage and doesn’t have a Hierarchal filesystem. That’s why when you create folder in the S3 console what you are really creating is a prefix meaning its just a placeholder to auto add the prefix string to the beginning of all objects. But all objects in the bucket are flat when it comes to the API. Plus the way the files are stored in the shards in S3. You can limit Throughput by treating prefixes like a hierarchical filesystem too much.
The more confusing thing I’ve had to explain to too many developers is that it’s not a directory regardless of what it looks like on the AWS website view. It’s a “prefix” meaning the entire thing, including the slashes, is just part of the filename, there is no real structure inside a bucket. So there’s no such thing as relative paths to search for things. When you call the list objects function with a prefix it’s the same as doing a “starts with” on a string. You can’t give it a partial “path” relative to another object that appears to be in another “folder”.
That’s mostly because it doesn’t have real directories, it just simulates them with string prefixes and if you forget that on large buckets you are going to have a bad time.
Imagine you have dir1/dir2 with “directories” inside, each one with millions of “files”/objects. When you try to “ls” the dir1/dir2 by querying Prefix='dir1/dir2/' and Delimiter='/' S3 will still scan all the millions of objects starting with the prefix, and then filter out the ones that have a / after the prefix.
There’s no benefit or drawback on it’s own. The way the keys and queries work are geared to S3’s nature as a key value object store - which requires it to support keys that wouldn’t be valid file paths or that have different delimiters.
Reminds me of the oilfield unit hell. If you even mention an SI unit on an offshore drilling platform, your beheaded carcass will be displayed on top of the derrick as a warning for others, left to be devoured by birds and the salty wind.
something very similar in mainframe land where they call files “datasets.” if you call a dataset a file, get ready to get an earfull!
which is ironic considering IBM themselves often call them files and one of the most popular dataset utilities is called FileAid. but if we acknowledge that, we lose a valuable opportunity to belittle and exclude newcomers
In AWS S3, they’re objects, not files. But you upload files to objects. And can download an object to a file. But they’re not files. Trust me bro.
You know, the same deal kind of applies about “losing newcomers”. Because I avooded using AWS because everything seemed cryptic and weird with its terminology. So other companies got my business.
Cryptic and weird and their documentation is the worst.
Awoooo!
There is a reason for that since S3 is not block storage and doesn’t have a Hierarchal filesystem. That’s why when you create folder in the S3 console what you are really creating is a prefix meaning its just a placeholder to auto add the prefix string to the beginning of all objects. But all objects in the bucket are flat when it comes to the API. Plus the way the files are stored in the shards in S3. You can limit Throughput by treating prefixes like a hierarchical filesystem too much.
The more confusing thing I’ve had to explain to too many developers is that it’s not a directory regardless of what it looks like on the AWS website view. It’s a “prefix” meaning the entire thing, including the slashes, is just part of the filename, there is no real structure inside a bucket. So there’s no such thing as relative paths to search for things. When you call the list objects function with a prefix it’s the same as doing a “starts with” on a string. You can’t give it a partial “path” relative to another object that appears to be in another “folder”.
That’s mostly because it doesn’t have real directories, it just simulates them with string prefixes and if you forget that on large buckets you are going to have a bad time.
What makes that different from a “real directory”?
Imagine you have
dir1/dir2with “directories” inside, each one with millions of “files”/objects. When you try to “ls” thedir1/dir2by queryingPrefix='dir1/dir2/'andDelimiter='/'S3 will still scan all the millions of objects starting with the prefix, and then filter out the ones that have a/after the prefix.And the benefit of this is …
There’s no benefit or drawback on it’s own. The way the keys and queries work are geared to S3’s nature as a key value object store - which requires it to support keys that wouldn’t be valid file paths or that have different delimiters.
In Magpiper, they’re Blarts.
deleted by creator
It’s called gatekeeping. You wouldn’t get it.
Reminds me of the oilfield unit hell. If you even mention an SI unit on an offshore drilling platform, your beheaded carcass will be displayed on top of the derrick as a warning for others, left to be devoured by birds and the salty wind.