Cluster File System and Storage

Tip

If you have data stored in an Amazon S3™ bucket, then you can use datastores in MATLAB to directly access the data without needing any storage on the cluster. For details, see Transfer Data to Amazon S3 Buckets and Access Data Using MATLAB. You can also select the following storage options when creating your cluster.

This page lists the different file storage options available for your MATLAB^® Parallel Server™ cluster on Amazon^® Web Services (AWS^®).

• Cluster Shared Storage

  • Shared Storage. A persistent shared file system storage distributed across the cluster. This file system also contains your MATLAB Job Scheduler database. To request persistent shared file storage that persists after you stop your cluster, select a storage size. The shared storage is mounted at /shared/persisted. When you stop the cluster, the content of /shared/persisted is retained as an EBS snapshot in your cloud account. When you delete the cluster, the content of /shared/persisted is permanently removed. For details, see the table below.

  • Temporary Storage. A temporary shared file system distributed across the cluster. To request temporary shared file storage, the headnode must contain ephemeral storage or AWS Instance Store which is available with 'd' instance types with various storage sizes. When the cluster starts, the storage is mounted at /shared/tmp. When you stop the cluster or if the cluster shuts down, the content of /shared/tmp is removed. Use it for temporary and intermediate data that you need to access from multiple worker nodes.

  • MATLAB Drive. To enable access to MATLAB Drive, you must use a personal cluster. The files available on MATLAB Drive are read-only. You can access the files in your MATLAB drive at the mounted location /MATLAB Drive by following steps at Access Files in Your MATLAB Drive (MATLAB).

• Local Machine Storage

  • Ephemeral Storage: This type of storage is available only on instance types that contain "d" in their name, for example, m5ad.24xlarge. Each ephemeral storage device (NVMe SSD) is mounted at /mnt/localnvmeN, where N goes from zero to the number of ephemeral storage devices minus one, for example, /mnt/localnvme1 corresponds to the second ephemeral storage device. Additional EBS volumes are mounted at /mnt/localebsN. The availability and size of instance storage vary based on the EC2 instance type chosen. Use this storage for temporary, instance-specific data processing but not for persistent data storage.

After selecting your storage options, click Create Cluster. For details on other cluster settings, see Create a Cloud Cluster.

All worker machines have access to local and cluster shared storage. You can use these folders for storing data generated by your jobs, and for data you want to transfer between the cluster and your client location. See Transfer Data to or from a Cloud Center Cluster. The paths are the same for all worker machines of the cluster. Changes to files and folders under /mnt/localebsN are not visible to other machines. Files and folders under the /shared mount point are shared by all worker machines of your cluster. Changes made by any machine are visible to all other machines. Each folder has different longevity, as shown in the table.

Location	Size	Usage
/MATLAB Drive	Depends on your MATLAB license. For more information, see MATLAB Drive™ Storage Quota (MATLAB).	Enabled when personal cluster is selected at cluster creation. A read-only file system on the worker machines.
/shared/persisted	Specified at cluster creation	The location of the cluster shared persisted storage and MATLAB Job Scheduler (MJS) data. This folder is shared among worker machines and is retained between cluster runs. Save data you want to retrieve on the next start of the cluster in folders and files under /shared/persisted. Since the content in /shared/persisted is retained when you stop or restart the cluster, the MJS data and history are preserved in this location between cluster runs. If /shared/persisted is not enabled for the cluster, MJS data and history are preserved in a separate data volume only in the headnode's local files ystem. Deleted when cluster is deleted.
/shared/tmp	Varies with instance type, only available if headnode is an instance with ephemeral storage (Instance Store)	This folder is shared among worker machines and is not retained between cluster runs. Use it to store temporary and intermediate data that must be visible or accessible from multiple worker machines. The available storage space depends on the ephemeral storage available on the selected machine instance type. Deleted when cluster is stopped.
/mnt/localebs0	100 GB	This mount point is created only if /shared/persisted is not enabled and it is available only on the headnode for MATLAB Job Scheduler data and history. The data is retained between cluster restarts.
/mnt/localnvmeN	Specified in cluster configuration	Only available and automatically enabled on instance types that contain "d" in their name, for example, m5ad.24xlarge. This storage volume is backed by ephemeral storage. Each ephemeral storage device (NVMe SSD) is mounted at /mnt/localnvmeN, where N goes from zero to the number of ephemeral storage devices minus one, for example, /mnt/localdata1 corresponds to the second ephemeral storage device.

Note:

• File sharing is enabled only among machines within the same cluster, no file sharing across machines from different clusters.

• You create, start, stop, and delete your cloud clusters independent of your local MATLAB session (MATLAB running from your local desktop or MATLAB session in MATLAB Online). Deleting an associated cluster object in MATLAB does not affect the cloud cluster or its storage.

• When a cluster times out, it shuts down and clears the contents of the root operating system volume, /shared/tmp, and /mnt/localdataN, but preserves the content of /shared/persisted. If you use an automatic shutdown setting for your cluster, ensure that you have all data you need from /shared/tmp, /mnt/localdata and /mnt/localdataN before this timeout occurs.

• The contents of /shared/tmp are built using ephemeral storage (AWS Instance Store).