#383 Uploading to Amazon S3 pro

Great episode. Can you explain what motivated you to build the final example without fog or carrierwave?

Great episode.

Can you please add in the show notes some of the libraries to process images client-side. It would be helpful! :)

Great screencast. I made an example app that uses http://blueimp.github.com/jQuery-File-Upload/ to upload straight to S3. It was using a rather complex iframe trick to get around Amazons same origin policy. That was before Amazon introduced CORS, now my app is basically obsolete... ;-)

Ryan, will you make a gem from the multifile uploader you showed in the end? I think that'd be a great idea.

Great cast Ryan! I used FOG recently via the Ruby Backup gem - to keep DB backups on S3, as well as a few different mirrors for user uploads. Great gem! Thanks again for the cast.

Great episode thx a lot.
I noticed something strange though. you create thumbnails based on the presence of the key attribute in the Painting class on the after_save

  def enqueue_image
    ImageWorker.perform_async(id, key) if key.present?
  end

but then you also set the key attribute in the perform method of the background task

    def perform(id, key)
      painting = Painting.find(id)
      painting.key = key #setting the key again
      painting.remote_image_url = painting.image.direct_fog_url(with_path: true)
      painting.save! # enqueue_image will get called again
      painting.update_column(:image_processed, true)
    end

I think this will result in putting another message in the background queue. which will always keep your worker busy recreating thumbnails. Am I correct?

How would you go about allowing file uploads to S3 from an API (e.g. I want to allow users of my site to upload images to my service through our API)?

I'd do an upload via fog but we're using Heroku and the connection times out after 30 seconds.

Nice episode Ryan, one thing, instead of using key.present? you should use: has_image_upload? since calling Painting.new.key will return uploads/<guid>/${filename} resulting in true when you call present?. It's documented in the README here: https://github.com/dwilkie/carrierwave_direct#uploading-from-a-remote-location

Ryan very timely as always. I was having trouble getting this to work in a legacy project. Same code worked great in a new rails app. But the legacy always resulted in "AccessDenied" response from AWS. Had me perplexed. Hours of tracing and debugging later, realized someone had set the time_zone in our rails config to EDT, and this threw off the expiration key in the policy_data that is sent in the POST request. Hope this might save others hours of tracing.

I got weird error. Have any of you experienced it?

InvalidPolicyDocumentInvalid Policy: Invalid JSON.06F63A9785323ECD+kEOXcGMeFmCmcwLDhJKI2xiUViqDMYjbrxwu9+b20mMG+ApNip7yEE0QNxjNuxK

There is a missing step here, and in the example code.

When uploading images to S3, you want to specify the Content-Type, otherwise S3 will just default it to "application/octet-stream", which can cause your browser to throw warnings when you retrieve the file.

This caught me out, because Content-Type is listed in the multi-part form request as a component of the "file" input, however crucially S3 will NOT look at that. You need to pass it again.

In my case, using jqueryFileUpload as Ryan did, I have

<input id="content-type" name="Content-Type" type="hidden" value="" />

and

add: (e,data)
  ...
  data.form.find('#content-type').val(file.type)
  data.submit()

I noticed that remote_image_url downloads the image from the remote server[S3] and the save method re-upload the images.

The implication of that:
If the server downloads the image and re-uploads it again, We did not spare anything.

Is there a way to upload just the process image (like the thumbnails) and not re-upload the source image?

You can setup EC2 instance or stick with carrierwave?
Ryan ... come on man ... how??, that's the question!!!

greetings from Palestine <3

I use the JQuery file upload (but just allowing one file to be selected). If the file is just part of a object, say user. So the user can upload a single picture to the user object. I don't want to create a new record when a file is uploaded, simply update a field. How does one accomplish that?

HI ryan, I am opening up your code straight from source file, how come am getting

"can't convert nil into String" ?

what do I need to do to change this?

thanks much

Can you add the Coffeescript to the show notes?

Anyboy got this working on IE?

What is the "key" attribute? It's neither a column in Paintings nor attr_accessor but we still can access it with Painting.first.key in the console. Pretty weird to me.

I had a "gotcha" with the S3Uploader class because my rails application always has a time zone set other than UTC and so my s3 request was failing with an invalid expiration time policy.

I just had to add the .utc to the end of this line in the S3Uploader intialize to get it working...

class S3Uploader
    def initialize(options)
      @options = options.reverse_merge(
        id: "fileupload",
        aws_access_key_id: ENV["AWS_ACCESS_KEY_ID"],
        aws_secret_access_key: ENV["AWS_SECRET_ACCESS_KEY"],
        bucket: ENV["AWS_S3_BUCKET"],
        acl: "public-read",
        expiration: 10.hours.from_now.utc,  #<--- added .utc
        max_file_size: 500.megabytes,
        as: "file"
      )
    end

Excellent episode Ryan. The community really appreciates your tireless efforts.

Excellent episode.I am able to upload single file with carrierwave.

But I got 403(Forbidden) error when I try to upload multiple files using jquery file upload.

When I try to upload single/multiple file I get alert box saying failed to upload and in console it says "Failed to load resource: the server responded with a status of 403 (Forbidden)" and "XMLHttpRequest cannot load https://xyz.s3.amazonaws.com/. Origin http://localhost:3000 is not allowed by Access-Control-Allow-Origin."

What do I need to do to fix this?

Thank you

I dare you to handle errors on direct uploads to S3. Then, the nightmare begins.

Hi,
just wondering if someone might be able to answer this?

1) how to prevent certain file type from being uploaded to AWS S3?
For example, i don't want user to upload video file at all, how to prevent it?

2) Is there a way in AWS S3 upload script to stop the files upload from triggering if each user is limit to 5 files upload per user?

Thanks for the help :- )

Hi,

Was wondering if CarrierWave and Fog be used to upload documents to box.com using a box API_KEY and and a box AUTH_TOKEN

Thanks

Hi,

I need files to be renamed before uploading
I get this to work :

    def key
      @key ||= "uploads/#{SecureRandom.hex}"
    end

But I need the extension at the end and this doesn't work:

    def key
      @key ||= "uploads/#{SecureRandom.hex}" + File.extname("${filename}")
    end

Thanks for the help

No matter what I do I get the XMLHttpRequest cannot load https://3r_photos.s3.amazonaws.com/. Origin http://localhost:3000 is not allowed by Access-Control-Allow-Origin.

error. If I tweak the env vars to be wrong, I still get the same error. I have CORS set as indicated. Not sure how to proceed. Any ideas on how to debug this? I've tried deleting and recreating a bucket. Next up is deleting my AWS account I think.

Noob question for the jQuery File Upload setup:
Should i be checking that the image url sent from AWS S3 after user upload is actually from my S3 bucket and not from somewhere else harmful?

How come when running the example code, I got the error can't convert nil into String. I am I already added the aws_access_key and aws_secret_key and bucket as well

I use the gem s3_direct_upload inspirated by the code, but I have a problem.

Upload and progress bar works perfectly, my problem is :

The url of the uploaded file is a part a a Medium I create. The Medium contain "name", "description", "file_url" etc.

I don't display the S3 form at the creation of my medium, after saving, I redirect_to the Edit action and then display the S3 form to add a file. I tried to set post:media_url(@medium) or post:medium_edit_path(@medium) but it doesn't work cos I guess update require PUT and not POST.

So, is it possible to implement PUT with the gem or may I do something different ?

Thanks for your help

Thanks Mark Wilson! that was my problem. Spent a day on it. I didn't see your comment yesterday! Thanks again

Anyone else having any issues similar to this after updating to 'jquery-rails 2.2.0'?

Uncaught Error: Syntax error, unrecognized expression: div class="upload"
img006.jpg
div class="progress"><div class="bar" style="width: 0%"...

When I select the files or file, nothing happens.

I get the following error on page load:

TypeError: document.getElementById(...) is null
[Break On This Error]

return document.getElementById(id).innerHTML;

The id is: template-upload when I check it in Firefox.

Any thoughts?

Im having the same problem

Hi Ryan :

I am using the source code for "Gallery After" folder. When I upload Image , the Images are getting uploaded to S3 bucket. But while displaying all the uploaded images, I am getting "Processing..." rather than the uploaded image.
What could have gone wrong ?

i needed to use 2 different classes to get this to work in my project. this was a good explanation but wouldn't work for me since I was already using carrierwave on a live site and had images uploaded to specific folders already on S3. What I did was created an uploader class ImageUploader using the CarrierWave_Direct gem like in this example, and then kept my original class but renamed it to ImageProcessor. This way the original uploaded file is saved in a different folder in S3 and then I don't have to change the folder structure for current photos on my site. you can check out my solution here if interested:

http://stackoverflow.com/questions/14791655/carrierwave-direct-gem-keep-current-directory-structure-on-s3/14842832#14842832

In your ASCIIcast path is "/config/carrierwave.rb", but definitely it should be "/config/initializers/carrierwave.rb"

great episode and I've got it (except I used carrierwave in the background to download from S3 and process it)

One small thing though: if the images have the same filename, (like if uploaded via iphone's mobile safari'), somehow the files end up being duplicates, instead of showing 2 different files. Has anyone had that issue?

When running the example code, I got the error:

can't convert nil into String

I have already added the aws_access_key and aws_secret_key and bucket as well. Please help me with this issue, I don't know where is the problem.

Ryan is awesome but the complexity of the solution is huge and I hate it.

for the custom s3 uploader, how would i go about adding attributes to be added along with the created photo. such as user_id and photo_name

I have downloaded the source code for the custom s3_uploader_form which uploads straight to s3.

Straight out of the box I can't upload anything to S3 as it always renders an alert stating "upload failed" for everything, however there is nothing in the trace to suggest why. Removing the paintings.js.coffee code and adding a submit_tag to the form enables me to upload files to s3, however this doesn't create a rails object because the js handles that. (I have another app running with carrierwave so I know these credentials, secret, access key etc are all correct)

My question is why are no files being uploaded when paintings.js.coffee is handling the file upload process?

I have a stackoverflow question about this question as well. http://stackoverflow.com/questions/16291848/upload-straight-to-s3-failing-railscasts-pro-383

thanks again for a great episode Ryan!

File chunking.

maxChunkSize: 1000000

I am finding that this directive is working for me. However S3 is not merging the file back together at their end.

I am only getting the last chunked bit of the file appearing in the folder.

I have found this

http://aws.amazon.com/articles/1109?_encoding=UTF8&jiveRedirect=1#14

But am a little unsure what to make of it. Was wondering if anyone else has come across this issue as it is pretty cool feature jquery upload. I am uploading video files and so being able to do it in batches means that their is less chance of file failure.

Hope someone can shed some light.

Kind regards,

Luke