#383 Uploading to Amazon S3 pro

Sep 26, 2012 | 14 minutes | Views, Ajax, Plugins

Here I show how to upload files directly to Amazon S3 using CarrierWave Direct and Fog. I then walk through another project which uses jQuery File Upload to handle multiple files and does not use CarrierWave.

Click to Play Video ▶

Download:
source codeProject Files in Zip (2.11 MB)
mp4Full Size H.264 Video (35.2 MB)
m4vSmaller H.264 Video (17.1 MB)
webmFull Size VP8 Video (21.6 MB)
ogvFull Size Theora Video (41 MB)

Martin Jarvis about 12 years ago

Great episode. Can you explain what motivated you to build the final example without fog or carrierwave?

vickash about 12 years ago

Probably because the carrierwave_directgem (and Amazon S3 itself), doesn't support uploading multiple files in a single POST request. You need some kind of client-side scripting to separate the files into individual POST requests and insert the other form fields (key, policy, signature etc.) on the fly each time.

The final example still uses Carrierwave though, just not for the upload. The jQuery plugin uploads the file straight to S3. Then, in the done callback, the URL of the uploaded file gets "calculated" and POST-ed to the paintings#create action. The URL comes back in the response header from S3 (as "location" I think), so you could probably get it from there if you wanted.

In the paintings#create action, things start to look normal again. Either in the app, or a worker process, the record gets created and Carrierwave copies the raw file from the S3 bucket, to the location where your Carrierwave settings want it to be, then runs any image processing you have set up.

BTW, having a separate bucket for holding the raw uploads from the jQuery plugin is probably a good idea if you're not persisting the record before letting the user upload. You can end up with files that never get associated to anything in the database.

Let Carrierwave copy each file that gets associated to a record and process it. Carrierwave won't delete the raw upload file, but you can use the aws-sdk gem to delete it when Carrierwave finishes working. Finally, set up a task to periodically clear out old files in the "raw upload bucket" and that should take care of cleanup.

For that last part, you might want to prepend a timestamp to the SecureRandom portion of the key, then use the prefix method in aws-sdk, so you can do something like raw_uploads_bucket.objects_with_prefix('20120926').delete_all, to delete all the raw uploads from yesterday. Might be a good idea to check that all those workers are finished running too.

If it works for your UX though, persisting the record before letting the user upload makes cleanup a bit simpler.

Renaud CUNY about 12 years ago

+1 on vickash explanation, the path you give seems pretty adapted to anyone who already has a running CarrierWave+Fog based app and wants to extend it to take benefit of CORS without too much refactoring.
Thanks a lot to both you and Ryan :).

Glenn Ericksen about 12 years ago

I've implemented something like this. I'm curious if anyone has had any luck getting it to work on IE8/9?

Romain Tribes about 12 years ago

Great episode.

Can you please add in the show notes some of the libraries to process images client-side. It would be helpful! :)

Wayne about 12 years ago

Looks like the jquery file upload plugin has the ability to process images client-side:

https://github.com/blueimp/jQuery-File-Upload/wiki/Upload-multiple-resolutions-of-one-image-with-multiple-resize-options

Clemens Kofler about 12 years ago

Yes, but as usual, it's not 100% cross browser compatible. It is supported by Chrome, current versions of Firefox and the newest version of Safari (see https://github.com/blueimp/jQuery-File-Upload/wiki/Browser-support at the bottom of the page).

Nico Ritsche about 12 years ago

Great screencast. I made an example app that uses http://blueimp.github.com/jQuery-File-Upload/ to upload straight to S3. It was using a rather complex iframe trick to get around Amazons same origin policy. That was before Amazon introduced CORS, now my app is basically obsolete... ;-)

Ryan, will you make a gem from the multifile uploader you showed in the end? I think that'd be a great idea.

christian rojas about 12 years ago

Ives Leutenegger about 12 years ago

Blaawhi about 12 years ago

+1!

Alisson Patrick about 12 years ago

Maciej Stańczyk about 12 years ago

Wayne about 12 years ago

I created a simple gem that extracts out the form helper and cofeescript found in Ryan's rails app he mentioned at the end. You can find it here:

https://github.com/BlissOfBeing/s3_direct_upload

There aren't many tests for it yet, but hey it works for now. Would be grateful for any pull requests ;)

Wayne about 12 years ago

The url has changed:

https://github.com/WayneHoover/s3_direct_upload

Glenn Ericksen about 12 years ago

Hey Wayne, very helpful. Do you know if your gem IE8/9 compatible?

Rutger Karlsson about 12 years ago

Got it working on IE8/9 ?

Wayne about 12 years ago

Working on IE8/9 support...

Wayne about 12 years ago

Just pushed an update that fixes IE8/9. For those that what to implement it as a fix to what Ryan provides its a rather small change to the helper method, see what I changed here:

https://github.com/waynehoover/s3_direct_upload/commit/b9d4e54e3a27780ec340afba8bf2df8d60e14990

Dan Weaver over 11 years ago

Thanks Wayne, really good gem. It's working great for me.

How would I use it when the upload objects are children another object?

I can't see a way to pass any kind of parent_id to the child on creation.

Tom Conroy about 12 years ago

Francis Brunelle about 12 years ago

AstonJ about 12 years ago

Great cast Ryan! I used FOG recently via the Ruby Backup gem - to keep DB backups on S3, as well as a few different mirrors for user uploads. Great gem! Thanks again for the cast.

Shady Sayed about 12 years ago

Great episode thx a lot.
I noticed something strange though. you create thumbnails based on the presence of the key attribute in the Painting class on the after_save

          ruby
        
  def enqueue_image
    ImageWorker.perform_async(id, key) if key.present?
  end

but then you also set the key attribute in the perform method of the background task

          ruby
        
    def perform(id, key)
      painting = Painting.find(id)
      painting.key = key #setting the key again
      painting.remote_image_url = painting.image.direct_fog_url(with_path: true)
      painting.save! # enqueue_image will get called again
      painting.update_column(:image_processed, true)
    end

I think this will result in putting another message in the background queue. which will always keep your worker busy recreating thumbnails. Am I correct?

mfcava about 12 years ago

I notice the same issue. Endless queue that kills my database performance.
What's the better way to avoid this?

Kai Schlichting almost 12 years ago

I played around with this issue, and the only solution that worked for me was adding an explicit flag to skip the processing:

          ruby
        
  def enqueue_image
    ImageWorker.perform_async(id, key) if key.present? && !skip_image_processing
  end

  def perform(id, key)
    painting = Painting.find(id)
    painting.key = key #setting the key again
    painting.remote_image_url = painting.image.direct_fog_url(with_path: true)
    painting.skip_image_processing = true
    painting.save! # enqueue_image will get called again
    painting.update_column(:image_processed, true)
  end

Any other, more elegant ideas?

Eric Chernuka over 11 years ago

A little late to the game, but I did this as an elegant solution.

          ruby
        
def enqueue_image
  ImageWorker.perform_async(id, key) if !image_processed && key.present?
end

Christian Gambardella over 11 years ago

Doesn't painting.image.recreate_versions! do the trick?

Tom Henrik Aadland almost 12 years ago

I just tested this app and can confirm your assumption. It kept on recreating the thumbnails.

Eric Boehs about 12 years ago

How would you go about allowing file uploads to S3 from an API (e.g. I want to allow users of my site to upload images to my service through our API)?

I'd do an upload via fog but we're using Heroku and the connection times out after 30 seconds.

Jefferson Carley about 12 years ago

Not sure if this is possible. The CORS configuration on S3 would have to allow origins from every single user that is using your API. The examples in this screencast upload files directly from a user's browser straight to S3. The file never goes through the rails application, only metadata about the file does.

You could get fancy and create some sort of registration process, where by the user of the API during the registration process enters the domain (origin) they plan to upload from. Or maybe they have a admin screen where they can change it out. AWS does expose an API where you can change settings on your S3 service. So when your users are in the admin screen, and they change the domain, then your rails app uses the S3 API to change the origin (or add) that the user entered. I hope I explained that well. I'm just spitballing here, never done this myself.

David Wilkie about 12 years ago

Nice episode Ryan, one thing, instead of using key.present? you should use: has_image_upload? since calling Painting.new.key will return uploads/<guid>/${filename} resulting in true when you call present?. It's documented in the README here: https://github.com/dwilkie/carrierwave_direct#uploading-from-a-remote-location

Matt O'Connell about 12 years ago

Ryan very timely as always. I was having trouble getting this to work in a legacy project. Same code worked great in a new rails app. But the legacy always resulted in "AccessDenied" response from AWS. Had me perplexed. Hours of tracing and debugging later, realized someone had set the time_zone in our rails config to EDT, and this threw off the expiration key in the policy_data that is sent in the POST request. Hope this might save others hours of tracing.

tknazn about 12 years ago

I got weird error. Have any of you experienced it?

InvalidPolicyDocumentInvalid Policy: Invalid JSON.06F63A9785323ECD+kEOXcGMeFmCmcwLDhJKI2xiUViqDMYjbrxwu9+b20mMG+ApNip7yEE0QNxjNuxK

Xifip about 12 years ago

I was getting the same and resolved it by including a redirect on success in the controller, as in the screencast.

@uploader.success_action_redirect = user_profile_url

Joseph Pawletko about 12 years ago

I had the same error, but my problem was not the redirect, but rather my Amazon S3 bucket parameter was incorrect, i.e., ENV["AWS_S3_BUCKET"] was not set to the correct value.

Brad Murray about 12 years ago

There is a missing step here, and in the example code.

When uploading images to S3, you want to specify the Content-Type, otherwise S3 will just default it to "application/octet-stream", which can cause your browser to throw warnings when you retrieve the file.

This caught me out, because Content-Type is listed in the multi-part form request as a component of the "file" input, however crucially S3 will NOT look at that. You need to pass it again.

In my case, using jqueryFileUpload as Ryan did, I have

<input id="content-type" name="Content-Type" type="hidden" value="" />

and

          
        
add: (e,data)
  ...
  data.form.find('#content-type').val(file.type)
  data.submit()

Rutger Karlsson about 12 years ago

Forgive a newbie. In ryans examples where do you put that code?

Chris Peters almost 12 years ago

app/assets/javascripts/paintings.js.coffee

chrishawk over 11 years ago

Thanks for that. I think this will fix an issue I'm having with IE10 playing audio on jPlayer from S3. One question though - where in the fields order do you put the content type field? I know order matters and I've tried every combination to no avail. Any thoughts?

Brett Dudo about 11 years ago

any further notes on this? I added 'content-type' => 'file' as a placeholder, and added the data.form.find('#content-type').val(file.type) to the coffee file above the data.submit()... nada. Images stop uploading.

but I, too, am getting the application/octet-stream warning...

Todd Quillen over 9 years ago

After a lot of time on google, I think I finally got this addressed:

add content_type to upload_helpber.rb (fields method and policy_data method):

          ruby
        
def fields
  {
    :key => key,
    :acl => @options[:acl],
    :policy => policy,
    :content_type => nil,
    :signature => signature,
    "AWSAccessKeyId" => @options[:aws_access_key_id],
  }
end

def policy_data
  {
    expiration: @options[:expiration],
    conditions: [
      ["starts-with", "$utf8", ""],
      ["starts-with", "$Content-Type", ""],
      ["starts-with", "$key", ""],
      ["content-length-range", 0, @options[:max_file_size]],
      {bucket: @options[:bucket]},
      {acl: @options[:acl]}
    ]
  }
end

add the two lines above "data.submit()" in the add method in the coffeescript file

          coffee
        
add: (e, data) ->
  ...
   $('#fileupload').append(data.context)
   data.form.find('#content_type').attr('name','Content-Type')
   data.form.find('#content_type').val(file.type)
   data.submit()

yossis about 12 years ago

I noticed that remote_image_url downloads the image from the remote server[S3] and the save method re-upload the images.

The implication of that:
If the server downloads the image and re-uploads it again, We did not spare anything.

Is there a way to upload just the process image (like the thumbnails) and not re-upload the source image?

The Man with Hocks about 12 years ago

You can setup EC2 instance or stick with carrierwave?
Ryan ... come on man ... how??, that's the question!!!

greetings from Palestine <3

Hisyam almost 12 years ago

Manners man.

Anyway if you want to process thumbnails using an EC2 instance, you can setup your own processing server using imgr or thumbd.

If you think setting up & maintaining a server purely for image processing is too much of a hassle, I suggest using external services like Blitline or thumbr.it. I've got a demo of Rails + Blitline working if you're interested.

Rutger Karlsson about 12 years ago

I use the JQuery file upload (but just allowing one file to be selected). If the file is just part of a object, say user. So the user can upload a single picture to the user object. I don't want to create a new record when a file is uploaded, simply update a field. How does one accomplish that?

Todd Quillen almost 11 years ago

Did you ever figure this out? I'm looking to do the same thing.

Todd Quillen over 10 years ago

I figured it out.

First, change the target for your form (leave the "post" though).

          
        
<div id="upload_form">

  <%= s3_uploader_form post: user_url(@user), as: "user[avatar]" do %>
    <%= file_field_tag :file, multiple: false %>
  <% end %>

  <script id="template-upload" type="text/x-tmpl">
    <div class="upload">
      {%=o.name%}
      <div class="progress"><div class="bar" style="width: 0%"></div></div>
    </div>
  </script>

</div>

In the javascript file (I converted it from coffee script) change this line in the done callback:

$.post(to, content);

          
        
jQuery.ajax({
  type: "PUT",
  url: to,
  data: content
});

awangstar about 12 years ago

HI ryan, I am opening up your code straight from source file, how come am getting

"can't convert nil into String" ?

what do I need to do to change this?

thanks much

tknazn about 12 years ago

I had that issue as well and you have to make sure aws_access_key_id , aws_secret_access_key, and bucket are not nil.

José Salinas over 11 years ago

Exactly! Thanks for the tip. Just restarting the server made it work.

Asher Cohen about 12 years ago

Can you add the Coffeescript to the show notes?

Rutger Karlsson about 12 years ago

Anyboy got this working on IE?

Giang Nguyen about 12 years ago

What is the "key" attribute? It's neither a column in Paintings nor attr_accessor but we still can access it with Painting.first.key in the console. Pretty weird to me.

Philipp about 12 years ago

The carrierwave-direct gem implements the key methods through calling mount_uploader in the painting model.

Mark Wilson about 12 years ago

I had a "gotcha" with the S3Uploader class because my rails application always has a time zone set other than UTC and so my s3 request was failing with an invalid expiration time policy.

I just had to add the .utc to the end of this line in the S3Uploader intialize to get it working...

          ruby
        
class S3Uploader
    def initialize(options)
      @options = options.reverse_merge(
        id: "fileupload",
        aws_access_key_id: ENV["AWS_ACCESS_KEY_ID"],
        aws_secret_access_key: ENV["AWS_SECRET_ACCESS_KEY"],
        bucket: ENV["AWS_S3_BUCKET"],
        acl: "public-read",
        expiration: 10.hours.from_now.utc,  #<--- added .utc
        max_file_size: 500.megabytes,
        as: "file"
      )
    end

Fabricio Pavano about 12 years ago

thank you! i had the same problem!

Michael Yagudaev over 11 years ago

Thank you! That was a very helpful comment. Ryan should add it to the ASCII cast too

Greg over 11 years ago

Rails has a method .httpdate, I think it's meant for this.

Chris Wise about 12 years ago

Excellent episode Ryan. The community really appreciates your tireless efforts.

sahil2628 almost 12 years ago

Excellent episode.I am able to upload single file with carrierwave.

But I got 403(Forbidden) error when I try to upload multiple files using jquery file upload.

When I try to upload single/multiple file I get alert box saying failed to upload and in console it says "Failed to load resource: the server responded with a status of 403 (Forbidden)" and "XMLHttpRequest cannot load https://xyz.s3.amazonaws.com/. Origin http://localhost:3000 is not allowed by Access-Control-Allow-Origin."

What do I need to do to fix this?

Thank you

Eric Boehs almost 12 years ago

It sounds like you don't have your CORS setup right (or at all). Check that you're using the right Origin Domain (as specified in the CORS policy).

Also you should use https://s3.amazonaws.com/xyz instead as you'll get invalid certs for https://xyz.s3.amazonaws.com.

Mateusz Sojka almost 12 years ago

I dare you to handle errors on direct uploads to S3. Then, the nightmare begins.

Thomas almost 12 years ago

Hi,
just wondering if someone might be able to answer this?

1) how to prevent certain file type from being uploaded to AWS S3?
For example, i don't want user to upload video file at all, how to prevent it?

2) Is there a way in AWS S3 upload script to stop the files upload from triggering if each user is limit to 5 files upload per user?

Thanks for the help :- )

Hani Atalla almost 12 years ago

Hi,

Was wondering if CarrierWave and Fog be used to upload documents to box.com using a box API_KEY and and a box AUTH_TOKEN

Thanks

idvix almost 12 years ago

Hi,

I need files to be renamed before uploading
I get this to work :

          ruby
        
    def key
      @key ||= "uploads/#{SecureRandom.hex}"
    end

But I need the extension at the end and this doesn't work:

          ruby
        
    def key
      @key ||= "uploads/#{SecureRandom.hex}" + File.extname("${filename}")
    end

Thanks for the help

303devworks almost 12 years ago

No matter what I do I get the XMLHttpRequest cannot load https://3r_photos.s3.amazonaws.com/. Origin http://localhost:3000 is not allowed by Access-Control-Allow-Origin.

error. If I tweak the env vars to be wrong, I still get the same error. I have CORS set as indicated. Not sure how to proceed. Any ideas on how to debug this? I've tried deleting and recreating a bucket. Next up is deleting my AWS account I think.

WhoDidThis almost 12 years ago

Noob question for the jQuery File Upload setup:
Should i be checking that the image url sent from AWS S3 after user upload is actually from my S3 bucket and not from somewhere else harmful?

Hoan Dang almost 12 years ago

How come when running the example code, I got the error can't convert nil into String. I am I already added the aws_access_key and aws_secret_key and bucket as well

taughtby almost 12 years ago

did you figure this out? Just got the same problem

SLB Conseils almost 12 years ago

I use the gem s3_direct_upload inspirated by the code, but I have a problem.

Upload and progress bar works perfectly, my problem is :

The url of the uploaded file is a part a a Medium I create. The Medium contain "name", "description", "file_url" etc.

I don't display the S3 form at the creation of my medium, after saving, I redirect_to the Edit action and then display the S3 form to add a file. I tried to set post:media_url(@medium) or post:medium_edit_path(@medium) but it doesn't work cos I guess update require PUT and not POST.

So, is it possible to implement PUT with the gem or may I do something different ?

Thanks for your help

Bobby Norton almost 12 years ago

Thanks Mark Wilson! that was my problem. Spent a day on it. I didn't see your comment yesterday! Thanks again

Andrew Kevin almost 12 years ago

Anyone else having any issues similar to this after updating to 'jquery-rails 2.2.0'?

Uncaught Error: Syntax error, unrecognized expression: div class="upload"
img006.jpg
div class="progress"><div class="bar" style="width: 0%"...

uchenna almost 12 years ago

i M HAVING SAME ISSUE

Demetris over 11 years ago

Have you found a solution to this problem ?

Daniele Spinosa over 11 years ago

You need to substitute this line

          javascript
        
data.context = $(tmpl("template-upload", file))`

with this other line in the initialize method of the paintings.js.cofee file

          javascript
        
data.context = $($.parseHTML(tmpl("template-upload", file))[1])

Shyam Visamsetty over 11 years ago

Thank you so much Daniele. Is this issue with new jquery version only.

Thanks,
Shyam
http://about.me/shyam_visamsetty

Brian almost 12 years ago

When I select the files or file, nothing happens.

I get the following error on page load:

TypeError: document.getElementById(...) is null
[Break On This Error]

return document.getElementById(id).innerHTML;

The id is: template-upload when I check it in Firefox.

Any thoughts?

Dainel Vera almost 12 years ago

Im having the same problem

Rajeev N Bharshetty almost 12 years ago

Hi Ryan :

I am using the source code for "Gallery After" folder. When I upload Image , the Images are getting uploaded to S3 bucket. But while displaying all the uploaded images, I am getting "Processing..." rather than the uploaded image.
What could have gone wrong ?

a5sk4s about 11 years ago

Most likely missed 'bundle exec sidekiq' like I did initially.

bitterloa almost 12 years ago

i needed to use 2 different classes to get this to work in my project. this was a good explanation but wouldn't work for me since I was already using carrierwave on a live site and had images uploaded to specific folders already on S3. What I did was created an uploader class ImageUploader using the CarrierWave_Direct gem like in this example, and then kept my original class but renamed it to ImageProcessor. This way the original uploaded file is saved in a different folder in S3 and then I don't have to change the folder structure for current photos on my site. you can check out my solution here if interested:

http://stackoverflow.com/questions/14791655/carrierwave-direct-gem-keep-current-directory-structure-on-s3/14842832#14842832

Lily Melnyk over 11 years ago

In your ASCIIcast path is "/config/carrierwave.rb", but definitely it should be "/config/initializers/carrierwave.rb"

Mark Kremer over 11 years ago

Thanks for reporting this mistake, I have corrected the text in the ASCIIcast

Boa Ho Man over 11 years ago

great episode and I've got it (except I used carrierwave in the background to download from S3 and process it)

One small thing though: if the images have the same filename, (like if uploaded via iphone's mobile safari'), somehow the files end up being duplicates, instead of showing 2 different files. Has anyone had that issue?

durandc about 9 years ago

@Boa Ho Man Did you ever figure out a solution to this? I can't get around this on the mobile web myself. If I upload more than one image file, only one of the images is uploaded multiple times.

Petr Šuška over 11 years ago

When running the example code, I got the error:

can't convert nil into String

I have already added the aws_access_key and aws_secret_key and bucket as well. Please help me with this issue, I don't know where is the problem.

Theo Bittencourt over 11 years ago

Check your initializer.

Virginian over 11 years ago

Try changing carrierwave.rb in your initializer to:

          ruby
        
CarrierWave.configure do |config|
  config.fog_credentials = {
    :provider               => 'AWS',       # required
    :aws_access_key_id      => 'xxx',       # required
    :aws_secret_access_key  => 'yyy',       # required
    :region                 => 'eu-west-1'  # required, defaults to 'us-east-1'
  }
  config.fog_directory  = 'name_of_your_aws_bucket' # required
  # see https://github.com/jnicklas/carrierwave#using-amazon-s3
  # for more optional configuration

end

Jiří Hradil over 11 years ago

Ryan is awesome but the complexity of the solution is huge and I hate it.

Los83 over 11 years ago

for the custom s3 uploader, how would i go about adding attributes to be added along with the created photo. such as user_id and photo_name

Brett Dudo about 11 years ago

for the user_id, you can do params[:photo][:user_id] = current_user.id or something like that. I'm still trying to figure out how to capture the photo name...

Andrew Rogers over 11 years ago

I have downloaded the source code for the custom s3_uploader_form which uploads straight to s3.

Straight out of the box I can't upload anything to S3 as it always renders an alert stating "upload failed" for everything, however there is nothing in the trace to suggest why. Removing the paintings.js.coffee code and adding a submit_tag to the form enables me to upload files to s3, however this doesn't create a rails object because the js handles that. (I have another app running with carrierwave so I know these credentials, secret, access key etc are all correct)

My question is why are no files being uploaded when paintings.js.coffee is handling the file upload process?

I have a stackoverflow question about this question as well. http://stackoverflow.com/questions/16291848/upload-straight-to-s3-failing-railscasts-pro-383

thanks again for a great episode Ryan!

lukebyrne over 11 years ago

File chunking.

maxChunkSize: 1000000

I am finding that this directive is working for me. However S3 is not merging the file back together at their end.

I am only getting the last chunked bit of the file appearing in the folder.

I have found this

http://aws.amazon.com/articles/1109?_encoding=UTF8&jiveRedirect=1#14

But am a little unsure what to make of it. Was wondering if anyone else has come across this issue as it is pretty cool feature jquery upload. I am uploading video files and so being able to do it in batches means that their is less chance of file failure.

Hope someone can shed some light.

Kind regards,

Luke

bparanj over 11 years ago

Simple direct File Upload to Amazon S3 using Javascript, jQuery, Amazon S3 CORS support, Simple Form with progress bar. Designed by me in Silicon Valley and developed by Denny: https://github.com/dennybritz or https://github.com/bparanj/s3-cors-upload-rails. Enjoy.

Lee McAlilly over 11 years ago

How would you write an integration test for this? I'm having trouble because it no longer redirects to the new action:

http://stackoverflow.com/questions/16993366/testing-carrierwave-file-uploads-to-s3-with-capybara-and-rspec

Lee McAlilly over 11 years ago

I figured this out if anyone else runs into this question. The key is using CarrierWaveDirect matchers:

http://stackoverflow.com/questions/16993366/testing-carrierwave-file-uploads-to-s3-with-capybara-and-rspec

gbenz over 11 years ago

Does anyone have an example that actually uses Jquery File Upload and then processes with carrierwave afterwards?

Looking for a tutorial / example for the final step that Ryan refers to here...

"One thing that this version of the application does not do is generate thumbnail versions of each image. Instead the full version of each image is shown scaled down. There are a variety of ways that we could add this functionality. We could set up an EC2 instance and have that process the images"

Eric Chernuka over 11 years ago

I'm also looking into this. Might just implement my own and I'll share my findings.

Grimmrobe about 11 years ago

Would be great to see if anybody has some examples of using carrierwave to process the uploaded file and ship the image versions back to s3.

Eric Chernuka over 11 years ago

This is a perfect use case for a project I'm starting. I'm running into a little issue where the sidekiq worker runs over and over on the same original image. I'm not quite sure whats going on, but if anyone can help why it keeps running that would be great. I haven't modified the 'gallery-after' code at all either.

          ruby
        
2013-07-23T16:38:10Z 36085 TID-ox5o30a8s INFO: Starting processing, hit Ctrl-C to stop
2013-07-23T16:39:01Z 36085 TID-ox5oe79dg Painting::ImageWorker JID-ee968f66ab665e06faa8215d INFO: start
2013-07-23T16:39:24Z 36085 TID-ox5oe79dg Painting::ImageWorker JID-ee968f66ab665e06faa8215d INFO: done: 22.813 sec
2013-07-23T16:39:24Z 36085 TID-ox5odqw08 Painting::ImageWorker JID-55da36d8eed3566f1dcdbb8c INFO: start
2013-07-23T16:39:47Z 36085 TID-ox5odqw08 Painting::ImageWorker JID-55da36d8eed3566f1dcdbb8c INFO: done: 22.585 sec
2013-07-23T16:39:47Z 36085 TID-ox5oe79dg Painting::ImageWorker JID-1adb5d111658cbe7eec89e6c INFO: start
2013-07-23T16:40:07Z 36085 TID-ox5oe79dg Painting::ImageWorker JID-1adb5d111658cbe7eec89e6c INFO: done: 20.703 sec
2013-07-23T16:40:07Z 36085 TID-ox5odqw08 Painting::ImageWorker JID-090912e3a2ea344478a4328c INFO: start
2013-07-23T16:40:31Z 36085 TID-ox5odqw08 Painting::ImageWorker JID-090912e3a2ea344478a4328c INFO: done: 24.182 sec
2013-07-23T16:40:31Z 36085 TID-ox5oe79dg Painting::ImageWorker JID-336203a55a260f03a70283d6 INFO: start
2013-07-23T16:40:47Z 36085 TID-ox5oe79dg Painting::ImageWorker JID-336203a55a260f03a70283d6 INFO: done: 15.941 sec
2013-07-23T16:40:47Z 36085 TID-ox5odqw08 Painting::ImageWorker JID-603b95db7f12733b4d2544e6 INFO: start
2013-07-23T16:41:01Z 36085 TID-ox5odqw08 Painting::ImageWorker JID-603b95db7f12733b4d2544e6 INFO: done: 13.247 sec
2013-07-23T16:41:01Z 36085 TID-ox5oe79dg Painting::ImageWorker JID-ace98b247b037499edc6474d INFO: start
2013-07-23T16:41:21Z 36085 TID-ox5oe79dg Painting::ImageWorker JID-ace98b247b037499edc6474d INFO: done: 20.634 sec
2013-07-23T16:41:21Z 36085 TID-ox5odqw08 Painting::ImageWorker JID-2598b2d38154f349b53fe6e5 INFO: start

Eric Chernuka over 11 years ago

even though the id is different its actually the same job running over and over.

Eric Chernuka over 11 years ago

Well, turns out that the after_save was causing the worker to spin up over and over. As I said above, the solution is to put this within your model.

          ruby
        
def enqueue_image
  ImageWorker.perform_async(id, key) if !image_processed && key.present?
end

Dan Weaver over 11 years ago

I'm using the example at the end of the video that just uses jquery-file-upload to upload direct to S3. It's all working fine but now I want to make the upload a child of a parent object.

I'm uploading files on the parent's new page. How do I pass the parent_id to the child object?

Furkan over 11 years ago

Is this safe to put these credentials on dom:

          html
        
<input name="policy" type="hidden" value="XXX">
<input name="signature" type="hidden" value="YYY">
<input name="AWSAccessKeyId" type="hidden" value="ZZZ">

Furkan over 11 years ago

and I get it from here: https://github.com/dwilkie/carrierwave_direct/issues/97

Mihai Banulescu over 11 years ago

Has anyone gotten the S3 direct upload version at the end with Rails 4? The uploading works, but the page doesn't get updated without a refresh, and I don't see the progress bars.

Perhaps it has something to do with Turbolinks?

Any help would be much appreciated.

Brett Dudo about 11 years ago

def create
  @album = Album.find(params[:album_id])
  params[:photo][:user_id] = current_user.id
  @photo = @album.photos.build(params[:photo])
  @photo.save
end

how do I capture params[:photo][:name]?

Matt Cowski about 11 years ago

What is {%=o.name%} ? I'm having issues on how to write it using Slim.

By the way, since I'm a noob, I found using gem 's3_direct_upload' to be very helpful. (THANK YOU!) https://github.com/WayneHoover/s3_direct_upload

But overall, I am getting the feeling that this solution isn't yet dependable for production apps because of browser compatibility and reading comments seems like there are many other bugs. So, in that case, is it better for me to use an external service i.e. inkfilepicker.com ? Other than $, are there any other cons of doing this?

Subrat Rout about 11 years ago

I do not want to use my AWS credentials to use directly in carrierwave.rb file as pushing it to Github will make it public. Is there a way to create a private file and call the credentials from there?

Preston Mueller about 11 years ago

You'll want to set the credentials in the environment as shown in the episode, so you can access them using ENV (and so the actual credentials aren't in the code).

As for how to manage the ENV variables, check out http://railscasts.com/episodes/85-yaml-configuration-revised or this gem https://github.com/laserlemon/figaro

Ken about 11 years ago

Since onbody update this episode,

[fog][WARNING] Unable to load the 'unf' gem. Your AWS strings may not be properly encoded.

If you got message like this, (You should, if you use the latest version fog) then you need

          ruby
        
gem 'unf'

check this for reference

Andris about 11 years ago

Using Rails 4.0.1 Carrierwave_direct solution. Images get uploaded to S3 but I'm geting an errors in the background processing:
"error_message"=>"Validation failed: Image could not download file: 403 Forbidden", "error_class"=>"ActiveRecord::RecordInvalid"

2013-11-14T16:56:09Z 3924 TID-owk35rblw WARN: Validation failed: Image could not download file: 403 Forbidden
2013-11-14T16:56:09Z 3924 TID-owk35rblw WARN: /Users/Andris/.rvm/gems/ruby-1.9.3-p392/gems/activerecord-4.0.1/lib/active_record/validations.rb:57:in `save!'

Andris about 11 years ago

When trying to debug, in ImageWorker method 'perform' product.image.direct_fog_url(with_path: true) is equal to:
"https://testbucket.s3.amazonaws.com/uploads/2b33ecda-2d3e-4962-b365-e55d07bae205/$%7Bfilename%7D"

Kris Quigley almost 11 years ago

Did you find a solution to this problem? I am having the same issue with 4.0.2

Eduardo almost 11 years ago

Did you solved your ploblem ?

Gaurav Tiwari over 10 years ago

Did you find any solution to this problem?

Alberto Harres Rocha about 11 years ago

I feel very frustrated after 2 weeks posting question at stack overflow, contacting people from this comment section and trying to figure out by myself how to resize images on the client side using Ryan Bate's method, unfortunately I couldn't solve, my project is 2 weeks late, and I'm very disappointed for not having the best solution explained here... Tha'ts my feedback from this screencast.

Stefanos Ioannou about 11 years ago

Sonic is frustrated

Denis almost 11 years ago

Oh i bet he is

Andre Bautista almost 11 years ago

As of right now I'm having trouble running this in production with Heroku. Locally I can upload a full-size/thumb-size version of the image to s3, however in production only a full-size image will be uploaded s3. To my understanding it's due to Heroku's support of background jobs, anyone have experience solving this?

John Trichereau almost 11 years ago

Same. This might be a stupid question but do we need to activate a worker process in order for RedisToGo to work with Sidekiq on Heroku?

rwwaskk about 10 years ago

have you solved this issue?

trbartel over 10 years ago

Great help Ryan! Since Ryan is creating new records with his upload, using POST from the INDEX page, has anyone got a PUT to work? I want to upload the file as a PUT/update after my record is created from the record's SHOW page. All my attempts still show "POST" in the logs when they fail. I have it working his way, but don't know what I'm missing this other way??

Yuttanant Suwansiri over 10 years ago

I've been getting "undefined method `fog_credentials=' for CarrierWave::Uploader::Base:Class" every time I tried to run rake db:reset or rails s. Could anyone please help?

rvgarimrj about 10 years ago

I have a doubt here. I have a profile model that has two image fields. Do I need to create two uploaders, one for each image field ? I can get this to work with carrierwave direct. Can anyone help me please ?

bparanj about 10 years ago

Simple File Upload using Amazon CORS support with progress bar

Nick Camillo about 10 years ago

godsend. thanks for sharing dude!

Jake Levine about 10 years ago

I had a lot of issues getting Fog to work correctly and fortunately just found this, which is more specific to the purpose here anyway:

https://github.com/sorentwo/carrierwave-aws

Hope that helps.

Denis about 10 years ago

oh man thanx so much for this

Wesley Vance over 8 years ago

This was a huge help... Thank you!