Script to access Amazon S3 files

I setup a cluster on AWS EC2 to run my scripts faster. Right now my script load input variables locally (on my C: drive) and write simulation results also locally. I setup an S3 bucket with a sub-directory for my input variables and a sub-directory for the results. How do I change the load and write function to point to the S3 bucket instead of my c: drive?

Thanks

Answer 1

[Jhon Wine]

I found datastore to be very easy to use. Run the code below in each worker.

 %Set S3 Credentials 
 setenv('AWS_ACCESS_KEY_ID', 'ID');
 setenv('AWS_SECRET_ACCESS_KEY','Key'); 
 setenv('AWS_REGION', 'us-west-2');
 %Load Data
 fp= 's3://mybucket/data/file.data';
 ds=fileDatastore(spectralFilePath,'ReadFcn',@AWSRead);
 data=ds.read;

Also define this function

 function data= AWSRead(fileName)
  fid = fopen(fileName);
  data= fread(fid,inf,'short');
  fclose(fid);
 end

Make sure the EC2 cloud is on the same site as S3 to avoid data transfer bottlenecks

Answer 2

[David Fink]

Since S3 uses a RESTful architecture, use 'webread' and 'webwrite' to access the S3 bucket.

webread: https://www.mathworks.com/help/matlab/ref/webread.html

webwrite: https://www.mathworks.com/help/matlab/ref/webwrite.html

[Jhon Wine]

Is there an example, how to use webread and webwrite with S3? There is no good tutorial..

Answer 3

[Harish Sundaresh]

can MATLAB read .json files from aws s3? I get these errors

%% webread

webread(fname)

Error using webread (line 119)

The 's3' protocol specified in URL, 'url', is not supported. Specify the URL with the protocol 'http://' or 'https://'.

%% jsondecode

Error using jsondecode

JSON syntax error: expected value but found end of text.

[Dmitry Kaplan]

I could not get webread to work. I am sure there is a way, but I could not figure it out.

Instead, I use something like this

fds = fileDatastore('s3://blah/blah/filename.json','FileExtensions',{'.json'},'readfcn',@fileread);  
file_data = read(fds);

This defines the read function for the datastore to be fileread(). Invoking the read() actually runs fileread() and passes to it the filename -- because the fds.ReadMode is 'file'.

[Stephane]

Thanks for sharing this Dmitry