18-11-2010

Java S3 upload using Spring RestTemplate

At Peecho, we use many of the AmazonAWS services. For example, we use EC2for our virtual machines and S3for all of our storage. Because of the scalable nature of S3,theoretically, we could serve an infinite amount of users uploading files to our platform without stressing our machines or infrastructureat all. The only drawback is that connected apps will have to uploadtheir files directly to S3 – which can be challenging at times. That’s why I’m writing this blog.First of all, the Spring RestTemplate classis awesome. It is a really neat and easy way to create requests to restful web services or even not so restful services. The cool thing is that you can configure marshallers on the template, which will automatically convert outgoing and incoming objects into XML, Json and more. For example, you can configure an XStreamMarshaller to marshall all outgoing objects into XML and all incoming XML into objects this way.Uploading to S3 can be really easy if you use one of the many libraries that Amazon provides for the different platforms likeJava, .Net, PHP etcetera. These libraries have easy-to-use methods to upload files to buckets, creating objects and setting policies. To make use of all this, you need an Amazon public and secret key, which is fine if you are uploading to your own S3 account. We need our customers to be able to upload to our S3 account and naturally we can’t give our customers the secret key to our amazon account because they could do all kinds of nasty evil stuff with it.Luckily, Amazon provides us with a way to upload files to S3 using a pre-signed url.This url contains a base64 encoded policy file, some paths to the data and a signed hash of the entire url – using your secret key. The policy files specify exactly what and where you can upload your data. For example, it specifies you can only upload *.jpg files to the/user-data/username/* path in S3. This file is generated on our server,using our secret key. This way customers of our API can only upload in directories that we specify and tampering with other customer’s files is impossible. Doing browser based uploads using a pre-signed url is explained in thisS3 article.Now, we have a signed url to post to and a valid policy file – but we still need to actually upload the data. This is where the rest template comes in. S3 expects a multi-part form post, instead of a normal file upload. Luckily there is a MessageConverterin Spring to create multi-part form posts! Configure it in yourapplication context like this:<bean id="restTemplate" class="org.springframework.web.client.RestTemplate"><property name="messageConverters"><list><bean class="org.springframework.http.converter.StringHttpMessageConverter" /><bean class="org.springframework.http.converter.FormHttpMessageConverter" /></list></property></bean>The FormHttpMessageConverter makesit possible to create a multi-part form post. In your Java code you cannow create the request:MultiValueMap<String, Object> form= new LinkedMultiValueMap<String, Object>(); form.add("key", objectKey);form.add("acl","private");form.add(« Content-Type », »image/jpeg »);form.add(« AWSAccessKeyId », awsAccessKeyId);form.add(« Policy »,serverGeneratedPolicy);form.add(« Signature »,serverGeneratedSignature);form.add(« Filename », «  »);form.add(« success_action_status », »201″);form.add(« file », new FileSystemResource(file));restTemplate.postForLocation(signedPutUrl,form); When just providing a map with only strings, the converter will convertit into a normal form post. However, when adding a file to the map, theconverter automatically makes it a multi-part form post. The Filenameparameter of the form is set to an empty string, which means amazon S3will use the filename of the uploaded file as the filename of the objectin S3.Well that is pretty much it – it can’t get much easier,right? 🙂

Jolien

Latest blogs

The 7th Magazine, 7 portals to inspire

Read blog

Illustrators Journal: covering the world of art

Read blog

Integrating the Peecho API and checkout: Issuu

Read blog