File bulk upload

Introduction

Uploading many small files is often slower than what could be achieved because we do not use the whole network bandwidth. Nextcloud has a bulk upload API where you can upload many small files together in order to optimize the use of network bandwidth.

Usage

The API is only available for registered users of your instance. And uses the path: <server>/remote.php/dav/bulk.

Starting a bulk upload

A bulk upload is simply using a request structured as HTTP multipart with related mime type.

Each file is then sent as one HTTP part.

Each file inside an HTTP Part will need the following headers:

• Content-Length: <file size>

• Content-Type: <mimetype>

• X-File-MD5: <md5 checksum>

• X-File-Mtime: <modification time of file>

• X-File-Path: <destination file path>

The reply is a json document with the following structure:

{
    "/small file.txt": {
        "error": false,
        "etag": "adb9aa24cbfa8e372c88431d1d99629a"
    }
}

Example of code to upload some test files with bulk upload protocol

#!/bin/bash

NB=$1
SIZE=$2

USER="admin"
PASS="admin"
SERVER="nextcloud.local"
UPLOAD_PATH="/tmp/bulk_upload_request_$(openssl rand --hex 8).txt"
BOUNDARY="boundary_$(openssl rand --hex 8)"
REMOTE_FOLDER="/test"

for ((i=1; i<="$NB"; i++))
do
        file_name=$(openssl rand --hex 8)
        file_local_path="./$file_name.txt"
        file_remote_path="$REMOTE_FOLDER/$file_name.txt"
        head -c "$SIZE" /dev/urandom > "$file_local_path"
        file_mtime=$(stat -c %Y "$file_local_path")
        file_hash=$(md5sum "$file_local_path" | awk '{ print $1 }')
        file_size=$(du -sb "$file_local_path" | awk '{ print $1 }')

        {
                echo -en "--$BOUNDARY\r\n"
                echo -en "X-File-Path: $file_remote_path\r\n"
                echo -en "X-OC-Mtime: $file_mtime\r\n"
                echo -en "X-File-Md5: $file_hash\r\n"
                echo -en "Content-Length: $file_size\r\n"
                echo -en "\r\n" >> "$UPLOAD_PATH"

                cat "$file_local_path"
                echo -en "\r\n" >> "$UPLOAD_PATH"
        } >> "$UPLOAD_PATH"
done

echo -en "--$BOUNDARY--\r\n" >> "$UPLOAD_PATH"

echo "Creating folder /test"
curl \
        -X MKCOL \
        -k \
        "https://$USER:$PASS@$SERVER/remote.php/dav/files/$USER/test" > /dev/null

echo "Uploading $NB files with total size: $(du -sh "$UPLOAD_PATH" | cut -d '   ' -f1)"
echo "Local file is: $UPLOAD_PATH"
curl \
        -X POST \
        -k \
        --progress-bar \
        --cookie "XDEBUG_PROFILE=true;path=/;" \
        -H "Content-Type: multipart/related; boundary=$BOUNDARY" \
        --data-binary "@$UPLOAD_PATH" \
        "https://$USER:$PASS@$SERVER/remote.php/dav/bulk"