What Is Chunked Upload?

Chunked upload is the process of breaking a file into smaller pieces, upload 'as is' and glue pieces into original file on the server side.The process includes several subsequent POST calls (transferring chunks), which mimics regular file upload mentioned in section above.

It sounds easy, but implementation can be quite tricky. In this article we will explain how to successfully implement a chunked file upload feature using Dropzone.js for front-end and pionl/laravel-chunk-upload package for the back-end.

Prerequisites

Before we start, make sure you meet necessary requirements:

• Clean Laravel 8.x installation
• with functional database
• with basic AUTH scaffolding system implemented (e,g, Bootstrap UI with AUTH)

NOTICE: Actually, we do not really need a database and auth routes for our exercise. We will also implement a basic media library feature by indexing our files. Here we try to achieve easy but meaningful tutorial.

Environment Setup

Let's install necessary package:

$ composer require pion/laravel-chunk-upload

...then:

$ php artisan vendor:publish --provider="Pion\Laravel\ChunkUpload\Providers\ChunkUploadServiceProvider"

In this line, we published config file for the package. You can visit it at ../config/chunk-upload.php

Create a symbolic link if you didn't already:

$ php artisan storage:link

TIP: If your inbuilt symlink() PHP function is disabled (as a server security measure) here is a quick override: $ ln -s /path_to_laravel_installation/storage/app/public /path_to_laravel_installation/public/storage

Now, visit Dropzone.js and download standalone archive from the page. Extract it anywhere on your PC and upload these two (dropzone.min.js, dropzone.min.css) files from DIST>MIN folder into ../public/vendor/dropzone/ of your Laravel installation. Generally, these two files are all we need. For more advanced use of Dropzone.js package, please consult documentation.

Building Routes, Controllers and Frontend

Ok, add routes to web.php file:

//Media library routes
Route::get('/medialibrary', [App\Http\Controllers\MediaLibraryController::class, 'mediaLibrary'])->name('media-library');

//FILE UPLOADS CONTROLER
Route::post('medialibrary/upload', [App\Http\Controllers\UploaderController::class, 'upload'])->name('file-upload');
Route::post('medialibrary/delete', [App\Http\Controllers\UploaderController::class, 'delete'])->name('file-delete');

Now, create MediaLibraryController:

$ php artisan make:controller MediaLibraryController

...and populate it:

<?php

namespace App\Http\Controllers;

use Illuminate\Http\Request;

class MediaLibraryController extends Controller
{
  /**
   * Create a new controller instance.
   *
   * @return void
   */
  public function __construct()
  {
    $this->middleware(['auth', 'verified']);
  }

  /**
   * Get Media Library page
   * @return View
   */
  public function mediaLibrary(Request $request){
    $user_obj = auth()->user();
    return view('medialibrary', ['user_obj' => $user_obj ]);
  }
}

...and UploaderController

$ php artisan make:controller UploaderController

...with code:

<?php

namespace App\Http\Controllers;

use Illuminate\Http\Request;
use Pion\Laravel\ChunkUpload\Exceptions\UploadFailedException;
use Storage;
use Illuminate\Http\UploadedFile;
use Pion\Laravel\ChunkUpload\Exceptions\UploadMissingFileException;
use Pion\Laravel\ChunkUpload\Handler\AbstractHandler;
use Pion\Laravel\ChunkUpload\Handler\HandlerFactory;
use Pion\Laravel\ChunkUpload\Receiver\FileReceiver;

class UploaderController extends Controller
{
  /**
   * Create a new controller instance.
   *
   * @return void
   */
  public function __construct()
  {
      $this->middleware(['auth', 'verified']);
  }

  /**
   * Handles the file upload
   *
   * @param Request $request
   *
   * @return JsonResponse
   *
   * @throws UploadMissingFileException
   * @throws UploadFailedException
   */
  public function upload(Request $request) {  //from web route
    // create the file receiver
    $receiver = new FileReceiver("file", $request, HandlerFactory::classFromRequest($request));

    // check if the upload is success, throw exception or return response you need
    if ($receiver->isUploaded() === false) {
      throw new UploadMissingFileException();
    }

    // receive the file
    $save = $receiver->receive();

    // check if the upload has finished (in chunk mode it will send smaller files)
    if ($save->isFinished()) {
      // save the file and return any response you need, current example uses `move` function. If you are
      // not using move, you need to manually delete the file by unlink($save->getFile()->getPathname())
      return $this->saveFile($save->getFile(), $request);
    }

    // we are in chunk mode, lets send the current progress
    /** @var AbstractHandler $handler */
    $handler = $save->handler();

    return response()->json([
      "done" => $handler->getPercentageDone(),
      'status' => true
    ]);
  }

  /**
   * Saves the file
   *
   * @param UploadedFile $file
   *
   * @return JsonResponse
   */
   protected function saveFile(UploadedFile $file, Request $request) {
     $user_obj = auth()->user();
     $fileName = $this->createFilename($file);

     // Get file mime type
     $mime_original = $file->getMimeType();
     $mime = str_replace('/', '-', $mime_original);

     $folderDATE = $request->dataDATE;

     $folder  = $folderDATE;
     $filePath = "public/upload/medialibrary/{$user_obj->id}/{$folder}/";
     $finalPath = storage_path("app/".$filePath);

     $fileSize = $file->getSize();
     // move the file name
     $file->move($finalPath, $fileName);

     $url_base = 'storage/upload/medialibrary/'.$user_obj->id."/{$folderDATE}/".$fileName;

     return response()->json([
      'path' => $filePath,
      'name' => $fileName,
      'mime_type' => $mime
     ]);
  }

  /**
   * Create unique filename for uploaded file
   * @param UploadedFile $file
   * @return string
   */
   protected function createFilename(UploadedFile $file) {
     $extension = $file->getClientOriginalExtension();
     $filename = str_replace(".".$extension, "", $file->getClientOriginalName()); // Filename without extension

     //delete timestamp from file name
     $temp_arr = explode('_', $filename);
     if ( isset($temp_arr[0]) ) unset($temp_arr[0]);
     $filename = implode('_', $temp_arr);

     //here you can manipulate with file name e.g. HASHED
     return $filename.".".$extension;
   }

  /**
   * Delete uploaded file WEB ROUTE
   * @param Request request
   * @return JsonResponse
   */
   public function delete (Request $request){

     $user_obj = auth()->user();

     $file = $request->filename;

     //delete timestamp from filename
     $temp_arr = explode('_', $file);
     if ( isset($temp_arr[0]) ) unset($temp_arr[0]);
     $file = implode('_', $temp_arr);

     $dir = $request->date;

     $filePath = "public/upload/medialibrary/{$user_obj->id}/{$dir}/";
     $finalPath = storage_path("app/".$filePath);

     if ( unlink($finalPath.$file) ){
       return response()->json([
         'status' => 'ok'
       ], 200);
     }
     else{
       return response()->json([
         'status' => 'error'
       ], 403);
     }
   }

}

Now, create /public/js/file_upload.js and populate it:

/**
 * GooTools.net custom JS for file upload
 */

Dropzone.options.datanodeupload =
  {
    parallelUploads: 1,  // since we're using a global 'currentFile', we could have issues if parallelUploads > 1, so we'll make it = 1
    maxFilesize: 1024,   // max individual file size 1024 MB
    chunking: true,      // enable chunking
    forceChunking: true, // forces chunking when file.size < chunkSize
    parallelChunkUploads: true, // allows chunks to be uploaded in parallel (this is independent of the parallelUploads option)
    chunkSize: 2000000,  // chunk size 2,000,000 bytes (~2MB)
    retryChunks: true,   // retry chunks on failure
    retryChunksLimit: 3, // retry maximum of 3 times (default is 3)
    renameFile: function(file) {
      var dt = new Date();
      var time = dt.getTime();
      return time+"_"+file.name;
    },
    acceptedFiles: ".jpeg,.jpg,.png,.txt",
    addRemoveLinks: true,
    timeout: 50000,
    removedfile: function(file) {
      var name = file.upload.filename;
      $.ajax({
        headers: {
          'X-CSRF-TOKEN': $('meta[name="csrf-token"]').attr('content')
        },
        type: 'POST',
        url: deleteAction,
        data: {
          filename: name,
          ts: generalTS,
          date: generalDATE,
        },
        success: function (data){
          console.log("File has been successfully removed!!");
        },
        error: function(e) {
          console.log(e);
        }});
        var fileRef;
        return (fileRef = file.previewElement) != null ?
          fileRef.parentNode.removeChild(file.previewElement) : void 0;
      },

      success: function(file, response)
        {
          console.log(response);
        },
      error: function(file, response)
        {
          return false;
        }
};

Create blade file /resources/views/medialibrary.blade.php with contents:

@extends('layouts.app')

@section('head_resources')

	@parent
	<link href="{{ asset('vendor/dropzone/dropzone.min.css') }}" rel="stylesheet">

@endsection

@section('content')
<div class="container-fluid">
	@include('modals.modal_upload')
	<div class="row justify-content-center">
		<div class="col-md-8">
			<div class="col-md-12 mb-4">
				<button type="button" class="btn btn-primary btn-ico" data-toggle="modal" data-target="#uploaderModal"><i class="fa fa-files-o"></i> {{ __('File Upload') }}</button>
			</div>
		</div>
	</div>
</div>
@endsection

@section('footer_resources')
	@parent
		<!-- Scripts -->
		<script src="{{ asset('vendor/dropzone/dropzone.min.js') }}" defer></script>
		<script src="{{ asset('js/file_upload.js') }}" defer></script>

		<script>
			var home_url = "{{env('APP_URL') }}";
			var deleteAction = '{{ route("file-delete") }}';
			var generalTS =  document.getElementById('dataTS').value;
			var generalDATE = document.getElementById('dataDATE').value;
			var token = '{!! csrf_token() !!}';
		</script>

@endsection

Then, edit /resources/views/layouts/app.blade.php in this way:

<!doctype html>
<html lang="{{ str_replace('_', '-', app()->getLocale()) }}">
<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">

    <!-- CSRF Token -->
    <meta name="csrf-token" content="{{ csrf_token() }}">

    <title>{{ config('app.name', 'Laravel') }}</title>

    @section('head_resources')
    <!-- Scripts -->
    <script src="{{ asset('js/app.js') }}" defer></script>

    <!-- Fonts -->
    <link rel="dns-prefetch" href="//fonts.gstatic.com">
    <link href="https://fonts.googleapis.com/css?family=Nunito" rel="stylesheet">

    <!-- Styles -->
    <link href="{{ asset('css/app.css') }}" rel="stylesheet">
    @show
</head>
<body>
    <div id="app">
        <nav class="navbar navbar-expand-md navbar-light bg-white shadow-sm">
            <div class="container">
                <a class="navbar-brand" href="{{ url('/') }}">
                    {{ config('app.name', 'Laravel') }}
                </a>
                <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarSupportedContent" aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="{{ __('Toggle navigation') }}">
                    <span class="navbar-toggler-icon"></span>
                </button>

                <div class="collapse navbar-collapse" id="navbarSupportedContent">
                    <!-- Left Side Of Navbar -->
                    <ul class="navbar-nav mr-auto">
                      <li class="nav-item">
                          <a class="nav-link" href="{{ route('media-library') }}">{{ __('Media Library') }}</a>
                      </li>
                    </ul>

                    <!-- Right Side Of Navbar -->
                    <ul class="navbar-nav ml-auto">
                        <!-- Authentication Links -->
                        @guest
                            @if (Route::has('login'))
                                <li class="nav-item">
                                    <a class="nav-link" href="{{ route('login') }}">{{ __('Login') }}</a>
                                </li>
                            @endif

                            @if (Route::has('register'))
                                <li class="nav-item">
                                    <a class="nav-link" href="{{ route('register') }}">{{ __('Register') }}</a>
                                </li>
                            @endif
                        @else
                            <li class="nav-item dropdown">
                                <a id="navbarDropdown" class="nav-link dropdown-toggle" href="#" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false" v-pre>
                                    {{ Auth::user()->name }}
                                </a>

                                <div class="dropdown-menu dropdown-menu-right" aria-labelledby="navbarDropdown">
                                    <a class="dropdown-item" href="{{ route('logout') }}"
                                       onclick="event.preventDefault();
                                                     document.getElementById('logout-form').submit();">
                                        {{ __('Logout') }}
                                    </a>

                                    <form id="logout-form" action="{{ route('logout') }}" method="POST" class="d-none">
                                        @csrf
                                    </form>
                                </div>
                            </li>
                        @endguest
                    </ul>
                </div>
            </div>
        </nav>

        <main class="py-4">
            @yield('content')
        </main>
    </div>
    @section('footer_resources')

    @show
</body>
</html>

Add blade template with upload form which opens modal upon pressing "File Upload" button /resources/views/modals/modal_upload.blade.php

<!-- Modal Upload-->
<?php
	$ts = time();
	$user_id = Auth::user()->id;
	$date = date("Y-m-d");
?>
<div class="modal fade" id="uploaderModal" tabindex="-1" role="dialog" aria-labelledby="uploadModalLabel" aria-hidden="true">
  <div class="modal-dialog modal-dialog-centered modal-lg" role="document">
    <div class="modal-content">
      <div class="modal-header">
        <h5 class="modal-title" id="uploadModalLabel">{{ __('Upload file') }}</h5>
        <button type="button" class="close" data-dismiss="modal" aria-label="Close">
          <span aria-hidden="true">&times;</span>
        </button>
      </div>
      <div class="modal-body">
				<div class="form-group row">
					<h5>{{ __('Drag and drop multipe files') }}</h5>
				</div>

      	<div id="uploaderHolder">
	      	<form action="{{ route('file-upload') }}"
	              class="dropzone"
	              id="datanodeupload">

	            <input type="file" name="file"  style="display: none;">
	            <input type="hidden" name="dataTS" id="dataTS" value="{{ $ts }}">
	            <input type="hidden" name="dataDATE" id="dataDATE" value="{{ $date }}">
	            @csrf
	        </form>
	    </div>

      </div>
      <div class="modal-footer">
        <button type="button" class="btn btn-primary" onClick="window.location.reload();">{{ __('Done') }}</button>
      </div>
    </div>
  </div>
</div>

Explanation:

In the MediaController we just an output page for web route and we will extend it later with Media model.

Considering our article topic, all magic happens in the UploaderController.php. Upload function starts receiving file in chunks with two potential successful outcomes: chunk received and last chunk received. Here Pionl package takes care of your file.

Once file upload is finished, we call saveFile function which handles file inside app storage. In our case, we use a pattern as storage/upload/medialibrary/__USER_ID__/__DATE__/.

At the front end, our model takes care of sending files. As you can see from the FORM tag, we bounded this form to the Dropzone script by a class name and form id which value is used to create nested object in the script. So, form id ("datanodeupload") value corresponds to the Dropzone.js nested object from the our file_upload.js file (Dropzone.options.datanodeupload). Considering Dropzone options we used, it is self explanatory with comments in the file itself.

Click here to see all available Dropzone.js options.

As we can see, inside our form we can send additional data to our back end. Additional data can be used for various purposes; in our case we send timestamp when page was sent to the browser and date. A date will be used to determine the name of the storage folder, since there is the chance for the last chunk to differ from the date of the first chunk. This approach is not mandatory.

Testing File Upload

Run your app, visit media library page, and open file upload modal. Try uploading several images (.png, .jpg) with at least one larger than a 2 MB (since we defined our chunks 2 MB large). If you did everything properly you should find uploaded files in /storage/app/public/upload/medialibrary/... folder.

Moreover, as your upload finishes, Dropzone offered us a remove link, which we setup in Dropzone options (addRemoveLinks and removedfile). UploaderController's delete function control process at the back end.

At this point, we concluded the chunked upload topic.

Media Library

The goal of media library is to sort user's uploaded files. We have two approaches here: by reading the user's storage folder, or to index uploaded files into the database. For the purpose of our exercise, we will use a database to store the information of the uploaded files.

First, let's create a Media migration table:

$ php artisan make:migration create_media_table

Fill with code:

<?php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

class CreateMediaTable extends Migration
{
    /**
     * Run the migrations.
     *
     * @return void
     */
    public function up()
    {
      Schema::create('media', function (Blueprint $table) {
          $table->id();
          $table->foreignId('user_id')
              ->constrained()
              ->onUpdate('cascade')
              ->onDelete('cascade');
          $table->string('name', 250);
          $table->string('mime', 150);
          $table->string('path', 250);
          $table->string('url', 250);
          $table->bigInteger('size');
          $table->timestamps();
      });
    }

    /**
     * Reverse the migrations.
     *
     * @return void
     */
    public function down()
    {
        Schema::dropIfExists('media');
    }
}

Create Media model:

$ php artisan make:model Media

<?php

namespace App\Models;

use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Database\Eloquent\Model;

class Media extends Model
{
  use HasFactory;

  protected $primaryKey = 'id';
  protected $table = 'media';
  protected $fillable = array('user_id', 'name', 'mime', 'path', 'url', 'size' );

  function user() {
    return $this->belongsTo( 'App\Models\User' );
  }

  /**
   * Convert bytes to more appropriate format e.g. MB,GB..
   * @param int $size
   * @return string
   */
  function humanFileSize() {
    if ($this->size >= 1073741824) {
      $fileSize = round($this->size / 1024 / 1024 / 1024,1) . 'GB';
    } elseif ($this->size >= 1048576) {
        $fileSize = round($this->size / 1024 / 1024,1) . 'MB';
    } elseif($this->size >= 1024) {
        $fileSize = round($this->size / 1024,1) . 'KB';
    } else {
        $fileSize = $this->size . ' bytes';
    }
    return $fileSize;
  }

}

Now, expand User.php model with function:

    function media() {
      return $this->hasMany( 'App\Models\Media', 'user_id' );
    }

Ok, we prerequisites needed to build our media library. The best way to do this is to place the code right after successful upload. So, let's edit UploaderControler.php:

//Add after namespace declaration
use App\Models\Media;

Then, add code inside saveFile() function right after return command:

   .
   .
   .
     $control_var = Media::create([
       'user_id' => $user_obj->id,
       'name' => $fileName,
       'mime' => $mime_original,
       'path' => $filePath,
       'url' => $url_base,
       'size' =>$fileSize
     ]);

if you haven't done already, update database tables:

$ php artisan migrate

If you have done everything properly, try to upload files after changes we made, you should see how the database is populated with media items.

Populate View With Media Items

In this section we will finish our media library view with a media items uploaded by user. Let's start to edit MediaLibraryController.php

//Add after namespace declaration
use App\Models\Media;

Then update our only route:

  public function mediaLibrary(Request $request){
    $user_obj = auth()->user();
    $media_obj = $user_obj->media->all();
    return view('medialibrary', ['user_obj' => $user_obj, 'media_obj' => $media_obj ]);
  }

...and replace 'content' section from the medialibrary.blade.php

@section('content')
<div class="container-fluid">
	@include('modals.modal_upload')
	<div class="row justify-content-center">
		<div class="col-md-8">
			<div class="col-md-12 mb-4">
				<button type="button" class="btn btn-primary btn-ico" data-toggle="modal" data-target="#uploaderModal"><i class="fa fa-files-o"></i> {{ __('File Upload') }}</button>
			</div>
			<div class="row">
				@foreach($media_obj as $media)
					<div class="col-md-2 border m-1 p-1">
						<img src="{!! env('APP_URL').'/'.$media->url !!}" style="width:100%; height:auto">
						<div>name: {{ $media->name }}</div>
						<div>mime: {{ $media->mime }}</div>
						<div>size: {{ $media->humanFileSize() }}</div>
					</div>
				@endforeach
			</div>
		</div>
	</div>
</div>
@endsection

Now, TEST IT!

Conclusion

In this article we explained how to implement large file upload in our Laravel application. Here we used a chunked upload technique with use of the two astonishing packages: Dropzone.js for front end and pionl/laravel-chunk-upload for back end.

Also, we introduced a media library feature with Media model as a natural extension to any kind of file upload activity: giving 'ownership' over the files in the web application logic.

Series Author: Aleksandar Milivojevic is a PHP programmer for over 10 years (Laravel and Wordpress). Writing articles in brief pauses between projects, to help out the community and popularize PHP in general.