Multipart Upload a large file to Oracle Cloud Infrastructure Object Storage using cURL

 When you have large files to be transferred either from on-premise or from another cloud to OCI, the best way is to do that is using Multipart upload. Multipart uploads help to accommodate objects that are too large for a single upload operation. OCI now supports doing multipart upload from cURL directly instead of using SDK’s or oci-cli. With a Pre-Authenticated URL (PAR) you can avoid configuring API keys and sharing it with external parties. PAR URL can provide a quick and secure way to upload files to OCI object storage.

The maximum size for an uploaded object is 10 TiB on OCI and object parts cannot be larger than 50 GiB. It is highly recommended to use multipart uploads to upload objects larger than 100 MiB. You can split the files into chunks and initiate a multipart upload from curl. You are responsible for creating the parts to upload. Object Storage just provides the API operations for the remaining steps. Once all the parts are uploaded, a final POST call is required to assemble the file. Let us now see what those steps are :

1. Create a PAR URL on Bucket

Press enter or click to view image in full size

PAR URL

Generated PAR URL:

https://objectstorage.us-ashburn-1.oraclecloud.com/p/OxZ47fcsRhQRxNqhPJzCpqvw23zqZQGV1L9nnLY-g3FzNgO223qo9mR77K2U1gpE/n/idmldytingzx/b/DWH-Load/o/

2. Split the large file into parts

You can split the file in parts using any OS utility. We will use split on Linux to divide the large file large_random_data.csv into 250MB chunks

Decide what part number you want to use for each part. Part numbers can range from 1 to 10,000. You do not need to assign contiguous numbers, but Object Storage constructs the object by ordering part numbers in ascending order.

split -b 250M -d large_random_data.csvls -ltrh x*-rw-rw-r — . 1 opc opc 250M Aug 2 22:56 x00
-rw-rw-r — . 1 opc opc 250M Aug 2 22:56 x01
-rw-rw-r — . 1 opc opc 250M Aug 2 22:56 x02
-rw-rw-r — . 1 opc opc 250M Aug 2 22:56 x03
-rw-rw-r — . 1 opc opc 250M Aug 2 22:56 x04
-rw-rw-r — . 1 opc opc 250M Aug 2 22:56 x05
-rw-rw-r — . 1 opc opc 250M Aug 2 22:56 x06
-rw-rw-r — . 1 opc opc 250M Aug 2 22:56 x07
-rw-rw-r — . 1 opc opc 50M Aug 2 22:56 x08

3. Create Multipart upload for the file on OCI Object Storage using the par URL

curl -X PUT -H “opc-multipart:true” https://objectstorage.us-ashburn-1.oraclecloud.com/p/OxZ47fcsRhQRxNqhPJzCpqvw23zqZQGV1L9nnLY-g3FzNgO223qo9mR77K2U1gpE/n/idmldytingzx/b/DWH-Load/o/large_random_data.csv

Output

{
 “namespace”: “idmldytingzx”,
 “bucket”: “DWH-Load”,
 “object”: “large_random_data.csv”,
 “uploadId”: “7f878479–00e1–4630–23b6–683deac88a78”,
 “timeCreated”: “2021–08–03T21:10:59.273Z”,
 “storageTier”: “Standard”,
 “accessUri”: “/p/OxZ47fcsRhQRxNqhPJzCpqvw23zqZQGV1L9nnLY-g3FzNgO223qo9mR77K2U1gpE/n/idmldytingzx/b/DWH-Load/u/large_random_data.csv/id/7f878479–00e1–4630–23b6–683deac88a78/”
}

Use the access URI together with the Object Storage hostname for the target region to upload parts, specifying the part number at the end of the URI. For example, to upload an object in 9 parts, issue the following PUT commands

curl -X PUT — data-binary ‘@x00’ https://objectstorage.us-ashburn-1.oraclecloud.com/p/OxZ47fcsRhQRxNqhPJzCpqvw23zqZQGV1L9nnLY-g3FzNgO223qo9mR77K2U1gpE/n/idmldytingzx/b/DWH-Load/u/large_random_data.csv/id/7f878479-00e1-4630-23b6-683deac88a78/1curl -X PUT — data-binary ‘@x01’ https://objectstorage.us-ashburn-1.oraclecloud.com/p/OxZ47fcsRhQRxNqhPJzCpqvw23zqZQGV1L9nnLY-g3FzNgO223qo9mR77K2U1gpE/n/idmldytingzx/b/DWH-Load/u/large_random_data.csv/id/7f878479-00e1-4630-23b6-683deac88a78/2curl -X PUT — data-binary ‘@x02’ https://objectstorage.us-ashburn-1.oraclecloud.com/p/OxZ47fcsRhQRxNqhPJzCpqvw23zqZQGV1L9nnLY-g3FzNgO223qo9mR77K2U1gpE/n/idmldytingzx/b/DWH-Load/u/large_random_data.csv/id/7f878479-00e1-4630-23b6-683deac88a78/3curl -X PUT — data-binary ‘@x03’ https://objectstorage.us-ashburn-1.oraclecloud.com/p/OxZ47fcsRhQRxNqhPJzCpqvw23zqZQGV1L9nnLY-g3FzNgO223qo9mR77K2U1gpE/n/idmldytingzx/b/DWH-Load/u/large_random_data.csv/id/7f878479-00e1-4630-23b6-683deac88a78/4curl -X PUT — data-binary ‘@x04’ https://objectstorage.us-ashburn-1.oraclecloud.com/p/OxZ47fcsRhQRxNqhPJzCpqvw23zqZQGV1L9nnLY-g3FzNgO223qo9mR77K2U1gpE/n/idmldytingzx/b/DWH-Load/u/large_random_data.csv/id/7f878479-00e1-4630-23b6-683deac88a78/5curl -X PUT — data-binary ‘@x05’ https://objectstorage.us-ashburn-1.oraclecloud.com/p/OxZ47fcsRhQRxNqhPJzCpqvw23zqZQGV1L9nnLY-g3FzNgO223qo9mR77K2U1gpE/n/idmldytingzx/b/DWH-Load/u/large_random_data.csv/id/7f878479-00e1-4630-23b6-683deac88a78/6curl -X PUT — data-binary ‘@x06’ https://objectstorage.us-ashburn-1.oraclecloud.com/p/OxZ47fcsRhQRxNqhPJzCpqvw23zqZQGV1L9nnLY-g3FzNgO223qo9mR77K2U1gpE/n/idmldytingzx/b/DWH-Load/u/large_random_data.csv/id/7f878479-00e1-4630-23b6-683deac88a78/7curl -X PUT — data-binary ‘@x07’ https://objectstorage.us-ashburn-1.oraclecloud.com/p/OxZ47fcsRhQRxNqhPJzCpqvw23zqZQGV1L9nnLY-g3FzNgO223qo9mR77K2U1gpE/n/idmldytingzx/b/DWH-Load/u/large_random_data.csv/id/7f878479-00e1-4630-23b6-683deac88a78/8curl -X PUT — data-binary ‘@x08’ https://objectstorage.us-ashburn-1.oraclecloud.com/p/OxZ47fcsRhQRxNqhPJzCpqvw23zqZQGV1L9nnLY-g3FzNgO223qo9mR77K2U1gpE/n/idmldytingzx/b/DWH-Load/u/large_random_data.csv/id/7f878479-00e1-4630-23b6-683deac88a78/9

4. To commit the multipart upload, use the POST command with the access URI. For example:

curl -X POST https://objectstorage.us-ashburn-1.oraclecloud.com/p/OxZ47fcsRhQRxNqhPJzCpqvw23zqZQGV1L9nnLY-g3FzNgO223qo9mR77K2U1gpE/n/idmldytingzx/b/DWH-Load/u/large_random_data.csv/id/7f878479-00e1-4630-23b6-683deac88a78/

You can delete all parts of an uncommitted or failed multipart upload using the DELETE command with the access URI. For example:

curl -X DELETE https://objectstorage.us-ashburn-1.oraclecloud.com/p/OxZ47fcsRhQRxNqhPJzCpqvw23zqZQGV1L9nnLY-g3FzNgO223qo9mR77K2U1gpE/n/idmldytingzx/b/DWH-Load/u/large_random_data.csv/id/7f878479-00e1-4630-23b6-683deac88a78/

5. To GET list of objects

curl -X get <unique-par-url>curl -X GET https://objectstorage.us-ashburn-1.oraclecloud.com/p/OxZ47fcsRhQRxNqhPJzCpqvw23zqZQGV1L9nnLY-g3FzNgO223qo9mR77K2U1gpE/n/idmldytingzx/b/DWH-Load/o/

Make sure you delete the par URL once the upload is complete or create it with a short validity for security purpose.

References :
__________

[1] https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/usingmultipartuploads.htm
[2] https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/usingpreauthenticatedrequests.htm