File Uploads

Uploading files to the server, handling and linking them

Description

File Upload is handled via the standard Servlet API.

YadaFramework offers some utility classes to more easily handle files after they have been uploaded:

storing files on disk
resizing images
assigning files to Entity objects
generating download links
managing the pool of uploaded files

This section shows how to upload files via plain form submission (ajax or not). For a way to upload by dragging, see Ajax upload by drag&drop.

Configuration

Maximum file size setting

You can set the maximum file size for an upload by means of the maxFileUploadSizeBytes configuration entry:

<config>
	<maxFileUploadSizeBytes>3000000</maxFileUploadSizeBytes>

The default is 50MB.

Maximum file size check

When the upload limit is reached, a message is logged with a debug severity. You can ensure to see the message with the following logback configuration:

<logger name="org.springframework.web.multipart.commons.YadaCommonsMultipartResolver" level="DEBUG"/>

A Request Attribute is also added to the Request with the name MaxUploadSizeExceededException and the MaxUploadSizeExceededException as a value. All Request Parameters sent with the form are lost.

The Java method

YadaCommonsMultipartResolver.limitExceeded(request)

can be used in a @Controller to take appropriate action when the file limit is exceeded, for example by returning an error:

if (YadaCommonsMultipartResolver.limitExceeded(request)) {
	yadaNotify.title("News not saved", model).error().message("File too big. Size limit is " + config.getMaxFileUploadSizeBytes()/(1024*1024) + " MB").add();
    return "/cms/news";
}

On an ajax POST, anything returned by the @Controller is discarded and the user only sees a generic error message.

The following may not be true anymore:

By default, Tomcat will also drop the connection and no response will be sent to the browser. This will result in a low-level error shown by the browser. The reason for this is explained here and can only avoided if you configure Tomcat not to drop the connection but keep uploading any excess data (maxSwallowSize="-1"). The drawback would be that network bandwidth would be wasted and the user will have to wait until the whole file is uploaded before being told that it was too big.

The best solution would be to check file size on the browser via javascript. This is not currently implemented in Yada but is something like the following:

function fileTooBig() {
    var file = $('input[name=attachment]')[0].files[0];
    if (file==null) {
            return false;
    }
    var sizeMega = file.size/1024/1024;
    if (sizeMega>3) {
            $('#fileTooBig').show();
            return true;
    }
    $('#fileTooBig').hide();
    return false;
}
$('input[name=attachment]').on('change', function() {
    fileTooBig();
});
$('#theForm').submit(function() {
    if (fileTooBig()) {
            event.preventDefault();
    }
});

Automate javascript checking of file size

HTML

File upload starts from a "multipart/form-data" form. This is a standard form with a input element of type "file":

<form method="POST" enctype="multipart/form-data" action="doUpload">
    File to upload: <input type="file" name="upfile"><br/>
    <button type="submit">Upload</button>
</form>

Form Fragment /yada/form/fileUpload

If you’re using a form backing bean you can include a yada fragment for the input tag. The following example also shows any error:

<form th:action="@{/profile}" th:object="${formProfile}" enctype="multipart/form-data" method="post"
th:classappend="${#fields.hasErrors('*')}? has-error" role="form">

    <div th:replace="/yada/form/fileUpload::field(fieldName='avatarImage',label='Avatar')"></div>

You can display a link to the uploaded file underneath the input field by passing an instance of YadaAttachedFile to the attachedFile fragment attribute. For other usage instructions see the source file for /yada/form/fileUpload.

JAVA

After submission, the uploaded file will be processed by Commons File Upload and sent to the @Controller as a MultipartFile object. You would normally add a field of that type to the form backing bean, but you can also handle it independently from the other form fields if you wish, by adding it to the @RequestMapping signature.

In the @Controller you have many options.

Just save the file

You can just save the file somewhere with YadaWebUtil.saveAttachment():

public String storeFile(MultipartFile submittedFile) {
    File destination = new File("someFolder");
    YadaWebUtil.saveAttachment(submittedFile, destination):

Then you will have to keep track of the file yourself somehow. The following sections show an alternative and more convenient way of dealing with file uploads.

YadaAttachedFile

Usually the uploaded file has to be associated to some Entity in the database: a user avatar or CV, the image of a product, the pdf for a trip. Use YadaAttachedFile to easily handle file attachments:

@Entity
public class Product {
    @OneToOne(cascade=CascadeType.PERSIST)
    protected YadaAttachedFile icon;

    @OneToOne(cascade=CascadeType.PERSIST)
    protected YadaAttachedFile specSheet;

After doing this you can make use of the functionality of YadaFileManager explained below. You shouldn’t use any cascade different from PERSIST or orphanRemoval annotations:

cascade SAVE would generate a ConcurrentModificationException when using the upload and crop workflow (images only - see below)
cascade REMOVE or orphanRemoval=true wouldn’t delete the file on disk
cascade PERSIST is needed when cloning the parent object (Product in the example above)

The YadaAttachedFile class stores some file-related information that you might want to keep:

the original name of the file uploaded by the user
the upload time
localized title and description
the folder where the file is stored
the name of three versions of the file: the original one and the ones scaled for desktop and mobile
the sort order relative to files of the same "group"
a "published" flag
a locale if the file has to be made available only to some specific locale. This could be useful for pdf files in different languages

YadaFileManager

Introduction

The YadaFileManager @Service is the single entry to all operations on uploaded files stored as YadaAttachedFile.

Every time a file is uploaded, it is stored in a folder named "uploads" in the <basePath> configured directory. This folder is created automatically if the tomcat process has enough permissions, otherwise you have to create it manually.

Saving the file

Every file is stored using the original file name. To prevent name duplicates a number is automatically appended at the end.

public String updateProfile(MultipartFile uploadedMultipart) {
    File uploadedFile = yadaFileManager.uploadFile(uploadedMultipart);

The File can then be attached to an Entity:

YadaAttachedFile newIcon = yadaFileManager.attachNew(uploadedFile, uploadedMultipart, "/userData", "icon");
if (newIcon!=null) {
    user.setIcon(newIcon);
    userRepository.save(user);
}

The yadaFileManager.uploadFile() call can be skipped when passing the MultipartFile directly to attachNew():

YadaAttachedFile newIcon = yadaFileManager.attachNew(uploadedMultipart, "/userData", "icon");

The association between the owning Entity and the new YadaAttachedFile instance is not created automatically by yadaFileManager.attachNew() and you have to do it explicitly as shown above. When the attach method is called, the original uploaded file is copied from the "uploads" folder into the target folder. The new file will have the new prefix specified and the YadaAttachedFile id at the end of the name. The original file is by default deleted from the "uploads" folder unless a specific configuration is set to false:

<yadaFileManager>
    <deleteUploads>false</deleteUploads>
</yadaFileManager>

Not deleting uploaded files allows the implementation of a filesystem-like feature where single files could be reused many times.

implement filesystem feature

In case you’re replacing a previous attachment, you only need to pass the previous YadaAttachedFile: the old files will be deleted and replaced with the new ones. No explicit database operation is needed in this case.

YadaAttachedFile previousIcon = user.getIcon();
YadaAttachedFile iconAttachedFile = yadaFileManager.attachReplace(previousIcon, uploadedFile, "icon", "jpg", null, null);

The difference between attachNew() and attachReplace() is that the former creates a new YadaAttachedFile instance each time and adds it to the database while the latter reuses the existing instance. If you use the attachNew method to replace an existing file, you will have to delete the old YadaAttachedFile object yourself so it’s better to use attachReplace in this scenario. AttachNew should be used on the first upload of a file or when an Entity can hold a list of files. There is no way to detect if you are using the wrong method, so be careful.

Complete Example

/**
 * Uploads an "icon" image for the user
 */
public String updateProfile(MultipartFile uploadedMultipart) {
	... fetch 'user' somehow ...
    if (uploadedMultipart!=null && !uploadedMultipart.isEmpty()) {
        YadaAttachedFile previousIcon = user.getIcon();
        if (previousIcon==null) {
            // Move the file to the "someFolder" directory and create a new YadaAttachedFile
            YadaAttachedFile newIcon = yadaFileManager.attachNew(uploadedMultipart, "/someFolder", "myprefix");
            if (newIcon!=null) {
                user.setIcon(newIcon);
                userRepository.save(user);
            }
        } else {
            // Replace the existing file with the uploaded one
            yadaFileManager.attachReplace(previousIcon, uploadedMultipart, "myprefix", "jpg", null, null);
        }
    }

Image variants

If the uploaded file is an image, it can be resized for desktop and mobile as needed by specifying the alternative dimensions:

yadaFileManager.attach(uploadedFile, "userData", "icon", "jpg", 1280, 768);

In the above example the image is converted to jpg and two additional versions are saved on disk. The conversion is performed with the command line tool configured in config/shell/resize (usually imagemagick).

To keep things simple, there are no high density versions for mobile: you should just use the desktop version.

link to the configuration section

File URL

In order to show images and allow file download, you need to add the relevant URL to the page. This is done by the methods YadaFileManager.getFileUrl(), YadaFileManager.getDesktopImageUrl(), YadaFileManager.getMobileImageUrl() that can either be used in the @Controller or directly in the HTML:

<img th:src="@{${@yadaFileManager.getDesktopImageUrl(user.icon)}}">
<a th:href="@{${@yadaFileManager.getFileUrl(product.manual)}}">Download manual</a>

If you call getMobileImageUrl() and a mobile image is not present, it will fall back to getDesktopImageUrl() which in turn falls back to getFileUrl().

Copy Files

When you duplicate an Entity you also need to duplicate the files on the filesystem using YadaFileManager.duplicateFiles() otherwise the new entity will reference the old files.

ConfiguratorShape clone = configuratorDao.copy(configuratorShape);
yadaFileManager.duplicateFiles(clone.getIcon());

This is not needed if the copy is done with YadaUtil.copyEntity() because the file on disk is also copied automatically.

Delete Files

Files can be removed from the filesystem with YadaFileManager.deleteFileAttachment(). All database objects must then be deleted manually.

YadaAttachedFile icon = user.getIcon();
yadaFileManager.deleteFileAttachment(icon);
user.setIcon(null); // Remove relationship before deletion
user = userDao.save(user);
yadaAttachedFileDao.delete(icon);

test that the above code works

Image upload and crop

Workflow

Usually images that users upload must be of a specific size and can be in (up to) two versions, one for desktop layout and another for mobile layout. Currently there is no specific image for tablet layout (use the desktop one) of for high density mobiles.

The upload form should specify the required size and should reject any smaller image. Bigger images should be allowed regardless of their proportions and should be cropped by the user if needed. Finally, the image has to be resized (reduced) to the target dimensions.

This is implemented by storing an instance of YadaCropQueue in the session, and starting a loop that asks the user to crop all images added to the queue until there are no more left.

Prerequisites

Imagemagick must be installed on the system.

Configuration

The required image size has to be configured in the conf.webapp.prod.xml file, as in the following example:

<config>
    <dimension targetImageExtension="jpg" preserveImageExtensions="gif">
        <news>
            <top>
                <desktop>1920,1200</desktop>
                <mobile>768,610</mobile>
                <pdf>3840,2400</pdf>
            </top>
            <thumbnail>
                <desktop>800,800</desktop>
                <mobile>400,400</mobile>
                <pdf>2000,2000</pdf>
            </thumbnail>
        </news>

targetImageExtension is the image format that all uploaded images will be converted to, unless specified in preserveImageExtensions which is a comma-separated list of extensions that should not be converted. This can be useful to preserve animated gifs. Then the desktop/mobile/PDF dimensions required for each image are specified, but all are optional. In this example there is one "news" image in three cropped sizes, one named "top" and another named "thumbnail". There’s no need to specify all the three dimensions (desktop/mobile/PDF), but at least one is required to make any sense of the crop operation. The above configuration can be read in your subclass of YadaConfiguration:

public YadaIntDimension[] getDimensionsNewsThumbnail() {
    return super.getImageDimensions("/news/thumbnail");
}

This will return an array of YadaIntDimension holding the desktop, mobile and PDF dimensions at position 0, 1 and 2, with a null value when the dimension has not been configured.

The command to crop and resize images must be specified in the configuration too. This example can crop and resize any image, preserving animated gifs if the gif extension has been included in the preserveImageExtensions attribute.

<config>
    <shell>
        <yadaCropAndResize timeoutseconds="20">
            <executable>convert</executable>
            <arg>${FILENAMEIN}</arg>
            <arg>-coalesce</arg>
            <arg>-repage</arg>
            <arg>0x0</arg>
            <arg>-crop</arg>
            <arg>${w}x${h}+${x}+${y}</arg>
            <arg>-resize</arg>
            <arg>${resizew}x${resizeh}&gt;</arg>
            <arg>+repage</arg>
            <arg>${FILENAMEOUT}</arg>
        </yadaCropAndResize>

This example works with any image but corrupts gif animations.

<yadaCropAndResize timeoutseconds="20">
    <executable>convert</executable>
    <arg>${FILENAMEIN}</arg>
    <arg>-background</arg> <!-- "-background white -flatten" converts any transparent png backround to white instead of the default black -->
    <arg>white</arg>
    <arg>-flatten</arg>
    <arg>-crop</arg>
    <arg>${w}x${h}+${x}+${y}</arg>
    <arg>-resize</arg>
    <arg>${resizew}x${resizeh}&gt;</arg>
    <arg>${FILENAMEOUT}</arg>
</yadaCropAndResize>

Be aware that the most recent version of imagemagick uses the "magick" command instead of "convert", which must become the first argument:

<executable>magick</executable>
<arg>convert</arg>
<arg>${FILENAMEIN}</arg>

For more details on shell command executions, see Shell Command Execution.

Java form bean

The easiest way to handle file uploads is to use the Entity Backing Beans technique. You need to add a @Transient field (with getter and setter) for each multipart file you need to receive:

@Entity
public class News implements CloneableDeep {
    @OneToOne(cascade=CascadeType.PERSIST)
    protected YadaAttachedFile thumbnail;

    @Transient
    private  MultipartFile thumbnailImage;

This allows for easy validation and handling of the uploaded file. You can also use a Form Backing Bean of course.

HTML form

The upload form can be as simple as a plain file input (here with spring/bootstrap5 validation added):

<form th:action="@{/addOrUpdateNews}" th:object="${news}" enctype="multipart/form-data"
	method="post" role="form" th:with="hasError=${#fields.hasErrors('myFieldName')}">
	<input type="file" name="myFieldName" accept="image/*" th:classappend="${hasError}?is-invalid">
	<div th:each="err : ${#fields.errors('myFieldName')}" th:text="${err}" class="invalid-feedback">Invalid image</div>

The form can also be implemented using the /yada/form/fileUpload fragment:

<form th:action="@{/addOrUpdateNews}" th:object="${news}" enctype="multipart/form-data" th:classappend="${#fields.hasErrors('*')}? has-error" method="post" role="form">
    <div th:replace="/yada/form/fileUpload::field(fieldName='thumbnailImage',size=${thumbnailSize},accept='image/*',label='Upload thumbnail image',required=${news.thumbnail==null},help='Thumbnail image',attachedFile=*{thumbnail})"></div>

These are the needed parameters:

fieldName: the name of the field in the backing bean that holds the multipart file
size: the YadaIntDimension taken from the configuration, using the biggest between desktop and mobile
'accept': should be used to allow the upload of image files only. If a non-image is uploaded, it wouldn’t pass validation anyway
required: should be false when the YadaAttachedFile is not null so that the user is not forced to upload the file when changing something else in the Entity
attachedFile: the YadaAttachedFile if you want to show a link to the image below the input field (optional)

Java Controller to show the form

When showing the form using the fragment example, the size model attribute must be set:

YadaIntDimension[] dimensionsDesktopAndMobile = config.getDimensionsNewsThumbnail();
YadaIntDimension biggestNeeded = YadaIntDimension.biggest(dimensionsDesktopAndMobile);
model.addAttribute("thumbnailSize", biggestNeeded);

Java Form submission

When the Controller receives the submitted data inside an instance of the Entity, the first thing is to check for the upload file size, then issue an error when the file is too big:

@RequestMapping("/addOrUpdateNews")
public String addOrUpdateNews(News news, BindingResult newsBinding, HttpServletRequest request, Model model, Locale locale) {
    if (YadaCommonsMultipartResolver.limitExceeded(request)) {
        yadaNotify.title("News not saved", model).error().message("File too big. Size limit is " + config.getMaxFileUploadSizeBytes()/(1024*1024) + " MB").add();
        return "/manager/news";
    }

If that check passes, the multipart should be extracted from the Entity because it won’t survive a save:

MultipartFile thumbnailImage = news.getThumbnailImage(); // Can be null

Next, the image size should be validated and when not big enough, the form should be returned with an error:

boolean valid = true;
YadaManagedFile thumbnailManagedFile = null;
YadaIntDimension[] thumbnailDimensionsDesktopMobile = null;
if (thumbnailImage!=null && !thumbnailImage.isEmpty()) {
    try {
        thumbnailDimensionsDesktopMobile = config.getDimensionsNewsThumbnail();
        YadaIntDimension biggestNeeded = YadaIntDimension.biggest(thumbnailDimensionsDesktopMobile);
        thumbnailManagedFile = yadaFileManager.manageFile(thumbnailImage);
        YadaIntDimension fileDimension = thumbnailManagedFile.getDimension();
        if (fileDimension.isUnset()) {
            newsBinding.rejectValue("thumbnailImage", "validation.value.invalidImage", "Invalid image file");
            valid = false;
        } else if (biggestNeeded.isAnyBiggerThan(fileDimension)) {
            newsBinding.rejectValue("thumbnailImage", "validation.value.smallImage", new Object[] {fileDimension, biggestNeeded}, "Image too small");
            valid = false;
        }
    } catch (IOException e) {
        log.error("Error uploading image", e);
        newsBinding.rejectValue("thumbnailImage", "dashboard.imageupload.error");
        valid = false;
    }
}

if (!valid) {
	yadaFileManager.delete(thumbnailManagedFile);
    return EDIT_VIEW;
}

The Entity should then be saved to store the new values, and the crop workflow can start. It is possible to sequentially crop as many images as there are in the form. Images to be cropped are stored in the session. It is important that, if the YadaSession object has been subclassed, it has the @Primary class annotation:

@Component
@Primary
@Scope(value="session", proxyMode=ScopedProxyMode.TARGET_CLASS)
public class ApplicationSession extends YadaSession<UserProfile> {

Back to the Controller, the validated image can be added to the crop queue:

boolean imageLoaded = false;
String cropRedirect = yadaWebUtil.redirectString("/manager/cropPage", locale);
String finalRedirect = yadaWebUtil.redirectString("/manager/journal", locale);
YadaCropQueue yadaCropQueue = applicationSession.addCropQueue(cropRedirect, finalRedirect); // Clear any previous abandoned crops and set the destination
if (thumbnailManagedFile!=null) {
    YadaCropImage yadaCropImage = yadaCropQueue.addCropImage(thumbnailManagedFile, thumbnailDimensionsDesktopMobile, FOLDER_NEWS, "thumb-");
    YadaAttachedFile newOrExisting = yadaCropImage.titleKey("crop.news.thumbnail").link(news.getThumbnail());
    news.setThumbnail(newOrExisting);
    imageLoaded=true;
}

The "/manager/cropPage" and "/manager/journal" strings are, respectively, the url where the crop page is located and the url where the user should land when all images in the queue have been cropped. If the YadaAttachedFile is modified outside the link method, it should be put back into the YadaCropImage otherwise you’ll get a "ConcurrentModificationException" after crop:

newOrExisting.setTitle(news.getTitle());
newOrExisting = yadaAttachedFileDao.save(newOrExisting);
yadaCropImage.setYadaAttachedFile(newOrExisting);

The final step is to redirect to the crop page:

if (!imageLoaded) {
    applicationSession.deleteCropQueue();
} else {
    news = newsRepository.save(news);
    log.debug("Entering crop workflow for news");
    return yadaCropQueue.getCropRedirect();
}

HTML Crop page

The crop page can be easily implemented by including the jcrop library and the yada imageCropper fragment:

<head>
    <link rel="stylesheet" th:href="@{/static/jcrop-3/jcrop.css}">
    <script th:src="@{/static/jcrop-3/jcrop.js}"></script>
</head>
<body class="yadaCropPage">
    <div class="container-fluid sec" th:with="cropQueue=${@applicationSession.cropQueue}, cropImage=${cropQueue.currentImage}">

        <h1><span th:text="#{${cropImage.titleKey}}">This is the title</span>
        	<span th:if="${cropQueue.totInitialImages>1}"> ([[#{crop.images.left(${cropQueue.count})}]])</span>
        </h1>
        <p>Drag the handles to the desired crop, then press the [[#{yada.crop.cropSubmit}]] button</p>

        <div th:replace="~{/yadacms/imageCropper::component(cropQueue=${cropQueue})}"></div>

    </div>
</body>

The actual crop of the image is already implemented in YadaMiscController so there’s nothing more to do. To post the form to a custom crop method instead, call the YadaCropQueue.setCropPerformAction().

Troubleshooting

The following exception: YadaInvalidUsageException: Concurrent modification on yadaAttachedFile. This happens if you set 'cascade=CascadeType.ALL' on the owning entity or if the yadaAttachedFile is merged after setting it on YadaCropImage

is thrown whenever the YadaAttachedFile inside YadaCropImage is different from the one found on db at the time of the final crop. This always happens in the following cases:

the Entity owning the YadaAttachedFile image has a cascade=SAVE on the attribute and it has been saved after calling yadaCropImage.link()
the YadaAttachedFile has been saved after calling yadaCropImage.link()

Solution: do not use the offending cascade or re-add the new version of YadaAttachedFile to the YadaCropImage:

yadaCropImage.setYadaAttachedFile(yadaAttachedFile);