View all AWS S3 buckets and list each buckets storage used
I have multiple AWS accounts and I need to list all S3 buckets per account and then view each buckets total size. Currently, I can only view the storage size of a single S3 bucket with:aws s3 ls...
View ArticleBest practices for external backups for S3 buckets?
I would like to know how everyone is making external backups of S3 buckets and best practices/recommendations for this. Regardless of how good AWS is it is still a single point of failure.Especially on...
View ArticleHow to make different permissions for different subdirectories of storage bucket
On Google Cloud Storage or Amazon S3, is there a way to give users a key to only a specific subdirectory of a storage bucket, for example:gs://images.eng.cm.com/avatars/<user-uuid>/xyzso the user...
View ArticleClik here to view.
Troubleshooting VPC flow logs with an S3 bucket using SSE-KMS encryption with...
I'm using Terraform and trying to set up automatic export of VPC flow logs into an S3 bucket in the same AWS account and region (ca-central-1) that has default encryption turned on with AWS-KMS (using...
View Articleauto delete aws S3 backups
I need to automate deletion of aws S3 backups ... evidently if I tag the backup with one of daily, weekly, monthly, yearly then aws will delete based on my desired retention counts per those periods...
View Articles3fs to mount S3 bucket with iamrole on non-aws machine
I am trying to mount S3 bucket on my on-premises CentOS-6 server using s3fs as below:[roots@myhost ~]# s3fs bucket-name /mnt/s3mnt -o...
View ArticleMigrating to AWS with Devops [closed]
What are they key stages of migrating to AWS with Devops ?Can anyone list a use case or workflow?
View ArticleAWS S3, as an organizer see empty bucket
I am an owner of organizer in on my aws account. When I enter S3 I see one folder, which is empty. But its being billed 8$ per month, so smth should be there.Is it possible that one account, of my...
View ArticleIs it possible to mount an s3 bucket as a point in a docker container?
I have a Java EE packaged as war file stored in an AWS s3 bucket. I will like to mount the folder containing the .war file as a point in my docker container. I have managed to do this on my local...
View ArticleAmazon S3 bucket does not redirect folder URLs to index.html
I have uploaded my HUGO static site into AWS S3 Bucket. I have also added the domain in cloudrfront.The home index.html page works fine, but I cannot access my resources.For example...
View ArticleClik here to view.
404 status code in response from Cloudfront for a single page react application
we have a single page app in ReactJS that is being served by CloudFront but the response status continues to be 404 for any nested pages (i.e pages other than the home page).How can this be resolved?...
View ArticleClik here to view.
Error: error loading plugins: loading plugin /data/ipfs/plugins/go-ds-s3.so:...
Aftercloning this reporun make build (getting as output s3plugin.so)run make installmake the appropriate changes in both /data/ipfs/config"Datastore": {"BloomFilterSize": 0,"GCPeriod":...
View ArticleWhat is the proper AWS IAM policy for access to a single bucket that allows...
I am working on a buildout where I am doing the following. Eventually I will need to make this scalable so I am not doing this manually each time.I have several clients dumping data into S3 buckets....
View ArticleHow can I grant access to my cloud storage for clients?
I am developing a webshop for large digital products (multiple .tif image files). These are currently stored in an S3 bucket.What is a recommended way to grant access to my customers once they have...
View ArticleHow can I efficiently scale a data lake?
I am running into problems with AWS Athena data lakes. They were initially very quick, but as we've scaled, they're performing poorly and are becoming expensive and inefficient. What can we do to solve...
View ArticleTerraform Aws S3 - deny all users except for a specific user
I have a bucket which I need to restrict to a specific user, I have written the following script but it still seems to allow all users to operate on the bucket.resource "aws_s3_bucket"...
View ArticleUse S3 To Store ConfigMap
I have a ConfigMap which stores Nginx configs and It is getting size exceeded. There are many site configs in there so I can't create multiple ConfigMaps.I thought of using S3 to store ConfigMap, is it...
View ArticleHow to assign an ACL for each S3 Bucket in a tuple/list using Terraform?
I have a terraform file set to create a list of S3 buckets, and I was assigning acl to each bucket the old way:resource "aws_s3_bucket" "this" { count = length(var.s3_bucket_names) bucket =...
View ArticleWhen OnPrem with Kubernetes, what is the recommended way to do file storage...
When you are running kubernetes on GKS, AKS, etc, you use your cloud provider specific storage bucket such as Amazon S3, google cloud storage, azure blob storage, to store your raw binary content in a...
View Article--- Article Not Found! ---
*** *** *** RSSing Note: Article is missing! We don't know where we put it!!. *** ***
View Articles3: reading parquet files with a subset of columns as argument; does this...
I have set of large tables stored as parquets on s3.In python, I'm using:pd.read_parquet(...,columns=columns)I'm reading the files directly from s3, without any database engine whatsoever for...
View ArticleHow can I use S3 as a filesystem
I'm not sure if it's something I'm doing wrong, or if the S3 web interface is just trash. There are a ton of basic capabilities that work in pretty much any other file storage product (Windows...
View ArticleHow to ascertain excesive use of my S3 bucket?
I have a S3 bucket that I use to store images for a website I manage. The bill is usually about $1/month and I store about 12.5 GB on the service. It's never really been an issue before, but recently...
View ArticleHow to design AWS infrastructure for my online course selling website?
I am designing an AWS architecture for hosting a website that sells online courses and merchandise products. For the purpose, I am planning to use the following architecture design:1 VPC1 Application...
View ArticleCreate an S3 blobstore on Nexus fails ("The S3 bucket exists but you are not...
Before mentioning my problem, I want to inform you that I'm using terraform and having nexus container running on EC2 orchestrated by Amazon ECS. Here is the objective: I want to create an S3 blobstore...
View ArticleHow to add a user to MinIO server using CLI?
I have MinIO server running in docker. I need to add a user from command linemc admin user add local ACCESSKEY SECRETKEYmc: <ERROR> Unable to add new user. Access Denied.why? How to add an user?I...
View Articlegranting access to personal folder in s3
I'm trying a setup where Cognito users are granted access only to personal folder inside a common bucket.Everything works as long as I use "aws:userid" as folder name,...
View ArticleCD with CodeDeploy and GitHub Actions - why specify S3 bucket?
Following tutorials like this one, I managed to set up a continuous deployment pipe to deploy automatically via AWS CodeDeploy to my EC2 instance using GitHub actions, triggered on merging (or rather...
View ArticleAzure DevOps release to AWS fails with bucket does not exist
We are building a simple website and pushing it to an S3 bucket on AWS with a release pipeline. We configured the service connection in ADO with an access key and secret from AWS and the deploy was...
View ArticleKubernetes vs Object Storage for static website
I'm evaluating the best approach for serving the static assets (JavaScript bundles, CSS, HTML, etc.) of a React application that uses Module Federation and Webpack. I have two potential...
View Article