#383 Uploading to Amazon S3 pro

Sep 26, 2012 | 14 minutes | Views, Ajax, Plugins

Here I show how to upload files directly to Amazon S3 using CarrierWave Direct and Fog. I then walk through another project which uses jQuery File Upload to handle multiple files and does not use CarrierWave.

Click to Play Video ▶

Download:
source codeProject Files in Zip (2.11 MB)
mp4Full Size H.264 Video (35.2 MB)
m4vSmaller H.264 Video (17.1 MB)
webmFull Size VP8 Video (21.6 MB)
ogvFull Size Theora Video (41 MB)

Martin Jarvis almost 13 years ago

Great episode. Can you explain what motivated you to build the final example without fog or carrierwave?

vickash almost 13 years ago

Probably because the carrierwave_directgem (and Amazon S3 itself), doesn't support uploading multiple files in a single POST request. You need some kind of client-side scripting to separate the files into individual POST requests and insert the other form fields (key, policy, signature etc.) on the fly each time.

The final example still uses Carrierwave though, just not for the upload. The jQuery plugin uploads the file straight to S3. Then, in the done callback, the URL of the uploaded file gets "calculated" and POST-ed to the paintings#create action. The URL comes back in the response header from S3 (as "location" I think), so you could probably get it from there if you wanted.

In the paintings#create action, things start to look normal again. Either in the app, or a worker process, the record gets created and Carrierwave copies the raw file from the S3 bucket, to the location where your Carrierwave settings want it to be, then runs any image processing you have set up.

BTW, having a separate bucket for holding the raw uploads from the jQuery plugin is probably a good idea if you're not persisting the record before letting the user upload. You can end up with files that never get associated to anything in the database.

Let Carrierwave copy each file that gets associated to a record and process it. Carrierwave won't delete the raw upload file, but you can use the aws-sdk gem to delete it when Carrierwave finishes working. Finally, set up a task to periodically clear out old files in the "raw upload bucket" and that should take care of cleanup.

For that last part, you might want to prepend a timestamp to the SecureRandom portion of the key, then use the prefix method in aws-sdk, so you can do something like raw_uploads_bucket.objects_with_prefix('20120926').delete_all, to delete all the raw uploads from yesterday. Might be a good idea to check that all those workers are finished running too.

If it works for your UX though, persisting the record before letting the user upload makes cleanup a bit simpler.

Renaud CUNY almost 13 years ago

+1 on vickash explanation, the path you give seems pretty adapted to anyone who already has a running CarrierWave+Fog based app and wants to extend it to take benefit of CORS without too much refactoring.
Thanks a lot to both you and Ryan :).

Glenn Ericksen almost 13 years ago

I've implemented something like this. I'm curious if anyone has had any luck getting it to work on IE8/9?

Romain Tribes almost 13 years ago

Great episode.

Can you please add in the show notes some of the libraries to process images client-side. It would be helpful! :)

Wayne almost 13 years ago

Looks like the jquery file upload plugin has the ability to process images client-side:

https://github.com/blueimp/jQuery-File-Upload/wiki/Upload-multiple-resolutions-of-one-image-with-multiple-resize-options

Clemens Kofler almost 13 years ago

Yes, but as usual, it's not 100% cross browser compatible. It is supported by Chrome, current versions of Firefox and the newest version of Safari (see https://github.com/blueimp/jQuery-File-Upload/wiki/Browser-support at the bottom of the page).

Nico Ritsche almost 13 years ago

Great screencast. I made an example app that uses http://blueimp.github.com/jQuery-File-Upload/ to upload straight to S3. It was using a rather complex iframe trick to get around Amazons same origin policy. That was before Amazon introduced CORS, now my app is basically obsolete... ;-)

Ryan, will you make a gem from the multifile uploader you showed in the end? I think that'd be a great idea.

christian rojas almost 13 years ago

Ives Leutenegger almost 13 years ago

Blaawhi almost 13 years ago

+1!

Alisson Patrick almost 13 years ago

Maciej Stańczyk almost 13 years ago

Wayne almost 13 years ago

I created a simple gem that extracts out the form helper and cofeescript found in Ryan's rails app he mentioned at the end. You can find it here:

https://github.com/BlissOfBeing/s3_direct_upload

There aren't many tests for it yet, but hey it works for now. Would be grateful for any pull requests ;)

Wayne almost 13 years ago

The url has changed:

https://github.com/WayneHoover/s3_direct_upload

Glenn Ericksen almost 13 years ago

Hey Wayne, very helpful. Do you know if your gem IE8/9 compatible?

Rutger Karlsson almost 13 years ago

Got it working on IE8/9 ?

Wayne almost 13 years ago

Working on IE8/9 support...

Wayne almost 13 years ago

Just pushed an update that fixes IE8/9. For those that what to implement it as a fix to what Ryan provides its a rather small change to the helper method, see what I changed here:

https://github.com/waynehoover/s3_direct_upload/commit/b9d4e54e3a27780ec340afba8bf2df8d60e14990

Dan Weaver about 12 years ago

Thanks Wayne, really good gem. It's working great for me.

How would I use it when the upload objects are children another object?

I can't see a way to pass any kind of parent_id to the child on creation.

Tom Conroy almost 13 years ago

Francis Brunelle over 12 years ago

AstonJ almost 13 years ago

Great cast Ryan! I used FOG recently via the Ruby Backup gem - to keep DB backups on S3, as well as a few different mirrors for user uploads. Great gem! Thanks again for the cast.

Shady Sayed almost 13 years ago

Great episode thx a lot.
I noticed something strange though. you create thumbnails based on the presence of the key attribute in the Painting class on the after_save

          ruby
        
  def enqueue_image
    ImageWorker.perform_async(id, key) if key.present?
  end

but then you also set the key attribute in the perform method of the background task

          ruby
        
    def perform(id, key)
      painting = Painting.find(id)
      painting.key = key #setting the key again
      painting.remote_image_url = painting.image.direct_fog_url(with_path: true)
      painting.save! # enqueue_image will get called again
      painting.update_column(:image_processed, true)
    end

I think this will result in putting another message in the background queue. which will always keep your worker busy recreating thumbnails. Am I correct?

mfcava almost 13 years ago

I notice the same issue. Endless queue that kills my database performance.
What's the better way to avoid this?

Kai Schlichting over 12 years ago

I played around with this issue, and the only solution that worked for me was adding an explicit flag to skip the processing:

          ruby
        
  def enqueue_image
    ImageWorker.perform_async(id, key) if key.present? && !skip_image_processing
  end

  def perform(id, key)
    painting = Painting.find(id)
    painting.key = key #setting the key again
    painting.remote_image_url = painting.image.direct_fog_url(with_path: true)
    painting.skip_image_processing = true
    painting.save! # enqueue_image will get called again
    painting.update_column(:image_processed, true)
  end

Any other, more elegant ideas?

Eric Chernuka about 12 years ago

A little late to the game, but I did this as an elegant solution.

          ruby
        
def enqueue_image
  ImageWorker.perform_async(id, key) if !image_processed && key.present?
end

Christian Gambardella about 12 years ago

Doesn't painting.image.recreate_versions! do the trick?

Tom Henrik Aadland over 12 years ago

I just tested this app and can confirm your assumption. It kept on recreating the thumbnails.

Eric Boehs almost 13 years ago

How would you go about allowing file uploads to S3 from an API (e.g. I want to allow users of my site to upload images to my service through our API)?

I'd do an upload via fog but we're using Heroku and the connection times out after 30 seconds.

Jefferson Carley almost 13 years ago

Not sure if this is possible. The CORS configuration on S3 would have to allow origins from every single user that is using your API. The examples in this screencast upload files directly from a user's browser straight to S3. The file never goes through the rails application, only metadata about the file does.

You could get fancy and create some sort of registration process, where by the user of the API during the registration process enters the domain (origin) they plan to upload from. Or maybe they have a admin screen where they can change it out. AWS does expose an API where you can change settings on your S3 service. So when your users are in the admin screen, and they change the domain, then your rails app uses the S3 API to change the origin (or add) that the user entered. I hope I explained that well. I'm just spitballing here, never done this myself.

David Wilkie almost 13 years ago

Nice episode Ryan, one thing, instead of using key.present? you should use: has_image_upload? since calling Painting.new.key will return uploads/<guid>/${filename} resulting in true when you call present?. It's documented in the README here: https://github.com/dwilkie/carrierwave_direct#uploading-from-a-remote-location

Matt O'Connell almost 13 years ago

Ryan very timely as always. I was having trouble getting this to work in a legacy project. Same code worked great in a new rails app. But the legacy always resulted in "AccessDenied" response from AWS. Had me perplexed. Hours of tracing and debugging later, realized someone had set the time_zone in our rails config to EDT, and this threw off the expiration key in the policy_data that is sent in the POST request. Hope this might save others hours of tracing.

tknazn almost 13 years ago

I got weird error. Have any of you experienced it?

InvalidPolicyDocumentInvalid Policy: Invalid JSON.06F63A9785323ECD+kEOXcGMeFmCmcwLDhJKI2xiUViqDMYjbrxwu9+b20mMG+ApNip7yEE0QNxjNuxK

Xifip almost 13 years ago

I was getting the same and resolved it by including a redirect on success in the controller, as in the screencast.

@uploader.success_action_redirect = user_profile_url

Joseph Pawletko almost 13 years ago

I had the same error, but my problem was not the redirect, but rather my Amazon S3 bucket parameter was incorrect, i.e., ENV["AWS_S3_BUCKET"] was not set to the correct value.

Brad Murray almost 13 years ago

There is a missing step here, and in the example code.

When uploading images to S3, you want to specify the Content-Type, otherwise S3 will just default it to "application/octet-stream", which can cause your browser to throw warnings when you retrieve the file.

This caught me out, because Content-Type is listed in the multi-part form request as a component of the "file" input, however crucially S3 will NOT look at that. You need to pass it again.

In my case, using jqueryFileUpload as Ryan did, I have

<input id="content-type" name="Content-Type" type="hidden" value="" />

and

          
        
add: (e,data)
  ...
  data.form.find('#content-type').val(file.type)
  data.submit()

Rutger Karlsson almost 13 years ago

Forgive a newbie. In ryans examples where do you put that code?

Chris Peters over 12 years ago

app/assets/javascripts/paintings.js.coffee

chrishawk about 12 years ago

Thanks for that. I think this will fix an issue I'm having with IE10 playing audio on jPlayer from S3. One question though - where in the fields order do you put the content type field? I know order matters and I've tried every combination to no avail. Any thoughts?

Brett Dudo almost 12 years ago

any further notes on this? I added 'content-type' => 'file' as a placeholder, and added the data.form.find('#content-type').val(file.type) to the coffee file above the data.submit()... nada. Images stop uploading.

but I, too, am getting the application/octet-stream warning...

Todd Quillen about 10 years ago

After a lot of time on google, I think I finally got this addressed:

add content_type to upload_helpber.rb (fields method and policy_data method):

          ruby
        
def fields
  {
    :key => key,
    :acl => @options[:acl],
    :policy => policy,
    :content_type => nil,
    :signature => signature,
    "AWSAccessKeyId" => @options[:aws_access_key_id],
  }
end

def policy_data
  {
    expiration: @options[:expiration],
    conditions: [
      ["starts-with", "$utf8", ""],
      ["starts-with", "$Content-Type", ""],
      ["starts-with", "$key", ""],
      ["content-length-range", 0, @options[:max_file_size]],
      {bucket: @options[:bucket]},
      {acl: @options[:acl]}
    ]
  }
end

add the two lines above "data.submit()" in the add method in the coffeescript file

          coffee
        
add: (e, data) ->
  ...
   $('#fileupload').append(data.context)
   data.form.find('#content_type').attr('name','Content-Type')
   data.form.find('#content_type').val(file.type)
   data.submit()

yossis almost 13 years ago

I noticed that remote_image_url downloads the image from the remote server[S3] and the save method re-upload the images.

The implication of that:
If the server downloads the image and re-uploads it again, We did not spare anything.

Is there a way to upload just the process image (like the thumbnails) and not re-upload the source image?

The Man with Hocks almost 13 years ago

You can setup EC2 instance or stick with carrierwave?
Ryan ... come on man ... how??, that's the question!!!

greetings from Palestine <3

Hisyam over 12 years ago

Manners man.

Anyway if you want to process thumbnails using an EC2 instance, you can setup your own processing server using imgr or thumbd.

If you think setting up & maintaining a server purely for image processing is too much of a hassle, I suggest using external services like Blitline or thumbr.it. I've got a demo of Rails + Blitline working if you're interested.

Rutger Karlsson almost 13 years ago

I use the JQuery file upload (but just allowing one file to be selected). If the file is just part of a object, say user. So the user can upload a single picture to the user object. I don't want to create a new record when a file is uploaded, simply update a field. How does one accomplish that?

Todd Quillen over 11 years ago

Did you ever figure this out? I'm looking to do the same thing.

Todd Quillen over 11 years ago

I figured it out.

First, change the target for your form (leave the "post" though).

          
        
<div id="upload_form">

  <%= s3_uploader_form post: user_url(@user), as: "user[avatar]" do %>
    <%= file_field_tag :file, multiple: false %>
  <% end %>

  <script id="template-upload" type="text/x-tmpl">
    <div class="upload">
      {%=o.name%}
      <div class="progress"><div class="bar" style="width: 0%"></div></div>
    </div>
  </script>

</div>

In the javascript file (I converted it from coffee script) change this line in the done callback:

$.post(to, content);

          
        
jQuery.ajax({
  type: "PUT",
  url: to,
  data: content
});

awangstar almost 13 years ago

HI ryan, I am opening up your code straight from source file, how come am getting

"can't convert nil into String" ?

what do I need to do to change this?

thanks much

tknazn almost 13 years ago

I had that issue as well and you have to make sure aws_access_key_id , aws_secret_access_key, and bucket are not nil.

José Salinas about 12 years ago

Exactly! Thanks for the tip. Just restarting the server made it work.

Asher Cohen almost 13 years ago

Can you add the Coffeescript to the show notes?

Rutger Karlsson almost 13 years ago

Anyboy got this working on IE?

Giang Nguyen almost 13 years ago

What is the "key" attribute? It's neither a column in Paintings nor attr_accessor but we still can access it with Painting.first.key in the console. Pretty weird to me.

Philipp almost 13 years ago

The carrierwave-direct gem implements the key methods through calling mount_uploader in the painting model.

Mark Wilson almost 13 years ago

I had a "gotcha" with the S3Uploader class because my rails application always has a time zone set other than UTC and so my s3 request was failing with an invalid expiration time policy.

I just had to add the .utc to the end of this line in the S3Uploader intialize to get it working...

          ruby
        
class S3Uploader
    def initialize(options)
      @options = options.reverse_merge(
        id: "fileupload",
        aws_access_key_id: ENV["AWS_ACCESS_KEY_ID"],
        aws_secret_access_key: ENV["AWS_SECRET_ACCESS_KEY"],
        bucket: ENV["AWS_S3_BUCKET"],
        acl: "public-read",
        expiration: 10.hours.from_now.utc,  #<--- added .utc
        max_file_size: 500.megabytes,
        as: "file"
      )
    end

Fabricio Pavano over 12 years ago

thank you! i had the same problem!

Michael Yagudaev over 12 years ago

Thank you! That was a very helpful comment. Ryan should add it to the ASCII cast too

Greg over 12 years ago

Rails has a method .httpdate, I think it's meant for this.

Chris Wise over 12 years ago

Excellent episode Ryan. The community really appreciates your tireless efforts.

sahil2628 over 12 years ago

Excellent episode.I am able to upload single file with carrierwave.

But I got 403(Forbidden) error when I try to upload multiple files using jquery file upload.

When I try to upload single/multiple file I get alert box saying failed to upload and in console it says "Failed to load resource: the server responded with a status of 403 (Forbidden)" and "XMLHttpRequest cannot load https://xyz.s3.amazonaws.com/. Origin http://localhost:3000 is not allowed by Access-Control-Allow-Origin."

What do I need to do to fix this?

Thank you

Eric Boehs over 12 years ago

It sounds like you don't have your CORS setup right (or at all). Check that you're using the right Origin Domain (as specified in the CORS policy).

Also you should use https://s3.amazonaws.com/xyz instead as you'll get invalid certs for https://xyz.s3.amazonaws.com.

Mateusz Sojka over 12 years ago

I dare you to handle errors on direct uploads to S3. Then, the nightmare begins.

Thomas over 12 years ago

Hi,
just wondering if someone might be able to answer this?

1) how to prevent certain file type from being uploaded to AWS S3?
For example, i don't want user to upload video file at all, how to prevent it?

2) Is there a way in AWS S3 upload script to stop the files upload from triggering if each user is limit to 5 files upload per user?

Thanks for the help :- )

Hani Atalla over 12 years ago

Hi,

Was wondering if CarrierWave and Fog be used to upload documents to box.com using a box API_KEY and and a box AUTH_TOKEN

Thanks

idvix over 12 years ago

Hi,

I need files to be renamed before uploading
I get this to work :

          ruby
        
    def key
      @key ||= "uploads/#{SecureRandom.hex}"
    end

But I need the extension at the end and this doesn't work:

          ruby
        
    def key
      @key ||= "uploads/#{SecureRandom.hex}" + File.extname("${filename}")
    end

Thanks for the help

303devworks over 12 years ago

No matter what I do I get the XMLHttpRequest cannot load https://3r_photos.s3.amazonaws.com/. Origin http://localhost:3000 is not allowed by Access-Control-Allow-Origin.

error. If I tweak the env vars to be wrong, I still get the same error. I have CORS set as indicated. Not sure how to proceed. Any ideas on how to debug this? I've tried deleting and recreating a bucket. Next up is deleting my AWS account I think.

WhoDidThis over 12 years ago

Noob question for the jQuery File Upload setup:
Should i be checking that the image url sent from AWS S3 after user upload is actually from my S3 bucket and not from somewhere else harmful?

Hoan Dang over 12 years ago

How come when running the example code, I got the error can't convert nil into String. I am I already added the aws_access_key and aws_secret_key and bucket as well

taughtby over 12 years ago

did you figure this out? Just got the same problem

SLB Conseils over 12 years ago

I use the gem s3_direct_upload inspirated by the code, but I have a problem.

Upload and progress bar works perfectly, my problem is :

The url of the uploaded file is a part a a Medium I create. The Medium contain "name", "description", "file_url" etc.

I don't display the S3 form at the creation of my medium, after saving, I redirect_to the Edit action and then display the S3 form to add a file. I tried to set post:media_url(@medium) or post:medium_edit_path(@medium) but it doesn't work cos I guess update require PUT and not POST.

So, is it possible to implement PUT with the gem or may I do something different ?

Thanks for your help

Bobby Norton over 12 years ago

Thanks Mark Wilson! that was my problem. Spent a day on it. I didn't see your comment yesterday! Thanks again

Andrew Kevin over 12 years ago

Anyone else having any issues similar to this after updating to 'jquery-rails 2.2.0'?

Uncaught Error: Syntax error, unrecognized expression: div class="upload"
img006.jpg
div class="progress"><div class="bar" style="width: 0%"...

uchenna over 12 years ago

i M HAVING SAME ISSUE

Demetris over 12 years ago

Have you found a solution to this problem ?

Daniele Spinosa over 12 years ago

You need to substitute this line

          javascript
        
data.context = $(tmpl("template-upload", file))`

with this other line in the initialize method of the paintings.js.cofee file

          javascript
        
data.context = $($.parseHTML(tmpl("template-upload", file))[1])

Shyam Visamsetty about 12 years ago

Thank you so much Daniele. Is this issue with new jquery version only.

Thanks,
Shyam
http://about.me/shyam_visamsetty

Brian over 12 years ago

When I select the files or file, nothing happens.

I get the following error on page load:

TypeError: document.getElementById(...) is null
[Break On This Error]

return document.getElementById(id).innerHTML;

The id is: template-upload when I check it in Firefox.

Any thoughts?

Dainel Vera over 12 years ago

Im having the same problem

Rajeev N Bharshetty over 12 years ago

Hi Ryan :

I am using the source code for "Gallery After" folder. When I upload Image , the Images are getting uploaded to S3 bucket. But while displaying all the uploaded images, I am getting "Processing..." rather than the uploaded image.
What could have gone wrong ?

a5sk4s almost 12 years ago

Most likely missed 'bundle exec sidekiq' like I did initially.

bitterloa over 12 years ago

i needed to use 2 different classes to get this to work in my project. this was a good explanation but wouldn't work for me since I was already using carrierwave on a live site and had images uploaded to specific folders already on S3. What I did was created an uploader class ImageUploader using the CarrierWave_Direct gem like in this example, and then kept my original class but renamed it to ImageProcessor. This way the original uploaded file is saved in a different folder in S3 and then I don't have to change the folder structure for current photos on my site. you can check out my solution here if interested:

http://stackoverflow.com/questions/14791655/carrierwave-direct-gem-keep-current-directory-structure-on-s3/14842832#14842832

Lily Melnyk over 12 years ago

In your ASCIIcast path is "/config/carrierwave.rb", but definitely it should be "/config/initializers/carrierwave.rb"

Mark Kremer over 12 years ago

Thanks for reporting this mistake, I have corrected the text in the ASCIIcast

Boa Ho Man over 12 years ago

great episode and I've got it (except I used carrierwave in the background to download from S3 and process it)

One small thing though: if the images have the same filename, (like if uploaded via iphone's mobile safari'), somehow the files end up being duplicates, instead of showing 2 different files. Has anyone had that issue?

durandc almost 10 years ago

@Boa Ho Man Did you ever figure out a solution to this? I can't get around this on the mobile web myself. If I upload more than one image file, only one of the images is uploaded multiple times.

Petr Šuška over 12 years ago

When running the example code, I got the error:

can't convert nil into String

I have already added the aws_access_key and aws_secret_key and bucket as well. Please help me with this issue, I don't know where is the problem.

Theo Bittencourt over 12 years ago

Check your initializer.

Virginian over 12 years ago

Try changing carrierwave.rb in your initializer to:

          ruby
        
CarrierWave.configure do |config|
  config.fog_credentials = {
    :provider               => 'AWS',       # required
    :aws_access_key_id      => 'xxx',       # required
    :aws_secret_access_key  => 'yyy',       # required
    :region                 => 'eu-west-1'  # required, defaults to 'us-east-1'
  }
  config.fog_directory  = 'name_of_your_aws_bucket' # required
  # see https://github.com/jnicklas/carrierwave#using-amazon-s3
  # for more optional configuration

end