Lab 2 - Handling file upload and sending email

HTTP File upload & download

File upload

In traditional, we could upload file using HTML form element by specifying the enctype attribute to multipart/form-data together with method="post". However, Sails requires all <input type="file" /> elements placing at the end of a form. Meaning that, we could not mix <input type="file" /> with other <input> elements in a form without correct ordering.

Thus, here are two different examples. One is the traditional approach, another one uses HTML 5 File API reading the file content and put it in a hidden input element.

File Upload Example 1: Upload file using HTMLFormElement to server file system

In this example, we would store file path of the uploaded file in User model avatarPath attribute.

Let’s use your comp3047-ls05-2018 Sails project as example. Please clone the project and open it up in Visual Studio Code.

git clone https://github.com/HKBU-COMP3047/lab-5-your_github_username

Adding `avatarPath` attribute in `api/models/User.js`.

We add an avatarPath attribute to User model in order to store the uploaded file path.

avatarPath: {
	type: 'string'
},

Add a `upload` action in `api/controllers/UserController.js`

Then we add a controller action to handle the file upload.

//...
upload1: async function(req, res) {

	if (req.method == 'GET')
		return res.view('user/upload1');

	req.file('avatarfile').upload({maxBytes: 10000000}, async function whenDone(err, uploadedFiles) {
		if (err) { return res.serverError(err); }
		if (uploadedFiles.length === 0){ return res.badRequest('No file was uploaded'); }

		await User.update({username: req.session.username}, {
			avatarPath: uploadedFiles[0].fd
		});

		return res.ok('File uploaded.');
	});
},
//...

The req.file('avatarfile').upload(options, callback) function would handle <input type="file" name="avatarfile" /> file upload control.

Config a route to the upload action

As usual, we need to define a route to the controller action in order to reach it. Please add the following line to config/routes.js.

'/user/upload1': 'UserController.upload1',

Adding ejs file

In a form element like <form action="..." method="post">, application/x-www-form-urlencoded is the default enctype when not specified. Thus we need to specify it to multipart/form-data by adding the enctype attribute.

Please create views/user/upload1.ejs file with the following content.

<form action="/user/upload1" method="post" enctype="multipart/form-data">
	<input type="file" accept="image/*" name="avatarfile" />
	<input type="submit" />
</form>

File Upload Example 2: Upload file data to Database with `<input type="hidden">` in `<form>` element

In this example, we are going to store the file content in User model avatar attribute.

Adding `avatar` attribute in `api/models/User.js`.

avatar: {
	type: 'string'
},

Adding `upload2` Controller action

//...
upload2: async function(req, res) {

	if (req.method == 'GET')
		return res.view('user/upload2');

	console.log('req.body.agree = ' + req.body.agree);

	await User.update({username: req.session.username}, {
		avatar: req.body.User.avatar
	});
	
	return res.ok('File uploaded.');
},
//...

Config a route to the `upload2` action

As usual, we need to define a route to the controller action in order to reach it. Please add the following line to config/routes.js.

'/user/upload2': 'UserController.upload2',

Adding ejs file

In this example, we need not to specify the enctype attribute of the form element. Instead, we need to add a implement a function, handling onchange event of the file input control with HTML 5 File API.

Please create views/user/upload2.ejs file with the following content.

<form action="/user/upload2" method="post">
	<input type="file" accept="image/*" onchange="handleFile(this.files)" />
	<div id="preview"></div>
	<input type="submit" />
</div>
<script>
function handleFile(files) {

  const file = files[0];
  if (!file.type.startsWith('image/')) return;

  var preview = document.getElementById('preview');
  
  var reader = new FileReader();
  reader.onload = function(e) {
	preview.innerHTML = "";
  
	var img = document.createElement('img');
    img.src = e.target.result;
    img.style = "max-width:33vw";
    preview.appendChild(img);

	var input = document.createElement('input');
	input.type = 'hidden';
	input.name = 'User[avatar]';
	input.value = e.target.result;
	preview.appendChild(input);
  }
  reader.readAsDataURL(file);

}
</script>

File Upload Example 2.1: Convert file to Data URI and store in Database

We may like to convert a local file to Data URI for embedding in HTML. To do so, we could use datauri module.

https://www.npmjs.com/package/datauri

Install `datauri` module

npm install datauri --save

Changing `upload` action to store Data URI in `UserController`

Originally Please locate following lines in UserController, which storing the file path in avatarPath attribute.

await User.update({username: req.session.username}, {
	avatarPath: uploadedFiles[0].fd
});

We modify it to store Data URI in avatar attribute by reading the file as Data URI.

const  DataURI  =  require('datauri').sync;
await User.update({username: req.session.username}, {
	avatarPath: uploadedFiles[0].fd,
	avatar: DataURI(uploadedFiles[0].fd)
});

Dynamic File download in Sails

In the previous examples, we have stored the file content in server file system or in database. Thus, we are going to implement the download function for both cases in this section.

File Download Example 1: Stream file content from server file system

Adding `avatar` Controller action

Please add the following avatar action to api/controllers/UserController.js.

avatar: async function(req, res) {
	var user = await User.findOne({username: req.params.username});
	if (!user || !user.avatarPath)
		return res.notFound();
	
	var SkipperDisk = require('skipper-disk');
	var fileAdapter = SkipperDisk();
	// set the filename
	// res.set("Content-disposition", "attachment; filename='avatar.jpg'");
	res.set('Content-type', 'image/jpeg');
	// Stream the file
	fileAdapter.read(user.avatarPath).on('error', function (err){
		return res.serverError(err);
	}).pipe(res);
},

Adding route to controller action

As usual, we need to define a route to the controller action in order to reach it. Please add the following line to config/routes.js.

'GET /user/:username/avatar.jpg': 'UserController.avatar',

After that, you could view the image by http://localhost:1337/user/admin/avatar.jpg, if you uploaded avatar using admin user.

Usage

<img src="/user/admin/avatar.jpg" />

File Download Example 2: Embed file content Data URI in HTML element

Adding a controller action

Please add the following profile action to api/controllers/UserController.js.

profile: async function(req, res) {
	var user = await User.findOne({username: req.session.username});
	if (!user) return res.notFound();

	return res.view('user/profile',{model: user});
},

Adding the route

'GET /user/profile': 'UserController.profile',

Adding view ejs file

Please create view/user/profile.ejs with the following code.

<h1><%=model.username%></h1>
<% if(model.avatar) { %>
<img src="<%=model.avatar%>" style="max-width:33vw" />
<a download="avatar.jpg" href="<%= model.avatar%>" target="_blank">Download avatar</a>
<% } %>

For the file we stored its Data URI in mode, we could simply use it in src attribute of an image element. If you would like to make it a download link, you could use <a href> element with download="filename" attribute like <a href="..." download="filename">.

Processing Excel file

Microsoft Excel is a very popular spreadsheet program in workplace. It would be great if our system could support reading and exporting such file format. In this section, we are going to parse and export XLSX file in Sails, and even in the browser.

SheetJS js-xlsx

SheetJS is a parser and writer library for various spreadsheet formats. It works on Nodejs and browsers, so that we could use it to process xlsx file in Sails controller or in Browser. Let’s add it to our project by npm install or npm i with --save option.

npm i --save xlsx

For the full documentation, please read it here.

Excel Example 1: Upload Excel file data to Database

Adding controller action

Here are codes for implementing the controller action. Please add it in api/controllers/PersonController.js. The code block for uploading file is similar to previous multipart/form-data example.

//...
import_xlsx: async function(req, res) {

	if (req.method == 'GET')
		return res.view('person/import_xlsx');

	req.file('file').upload({maxBytes: 10000000}, async function whenDone(err, uploadedFiles) {
		if (err) { return res.serverError(err); }
		if (uploadedFiles.length === 0){ return res.badRequest('No file was uploaded'); }

		var XLSX = require('xlsx');
		var workbook = XLSX.readFile(uploadedFiles[0].fd);
		var ws = workbook.Sheets[workbook.SheetNames[0]];
		var data = XLSX.utils.sheet_to_json(ws);
		console.log(data);
		var models = await Person.createEach(data).fetch();
		if (models.length == 0) {
			return res.badRequest("No data imported.");
		}
		return res.ok("Excel file imported.");
	});
},
//...

Adding ejs file of upload form

Please create views/person/import_xlsx.ejs file with the following content.

<form action="/person/import_xlsx" method="post" enctype="multipart/form-data">
	<input type="file" accept="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" name="file" />
	<input type="submit" />
</form>

Adding route to the controller action

As usual, we need to define a route to the controller action in order to reach it. Please add the following line to config/routes.js.

'/person/import_xlsx': 'PersonController.import_xlsx',

After that let’s upload a excel file by navigating to http://localhost:1337/person/import_xlsx. A sample excel file could be downloaded from the Moodle.

Excel Example 2: Export database as Excel file

In this example, we are going to implement a controller action to download all Person data as Excel file from the Database.

Create a controller action

Please add the following code in PersonController.

export_xlsx: async function(req, res) {

	var models = await Person.find();
	
	var XLSX = require('xlsx');
	var wb = XLSX.utils.book_new();
	
	var ws = XLSX.utils.json_to_sheet(models.map(model => {
		return {
			name:model.name,
			age:model.age
		}
	}));
	XLSX.utils.book_append_sheet(wb, ws, "Person");
	
	res.set("Content-disposition", "attachment; filename=person.xlsx");
	return res.end(XLSX.write(wb, {type:"buffer", bookType:"xlsx"}));
},

Setting route to the controller action

As usual, we need to define a route to the controller action in order to reach it. Please add the following line to config/routes.js.

'/person/export_xlsx': 'PersonController.export_xlsx',

Please try it out by going to http://localhost:1337/person/export_xlsx. It would download all Person data as person.xlsx.

Excel Example 3: Parsing XLSX file data in Browser with client-side Javascript

Adding `upload2` Controller action

//...
import_data: async function(req, res) {
	if (req.method == 'GET')
		return res.view('person/import_data');

	if (typeof req.body.Persons === "undefined")
		return res.badRequest("Form-data not received.");
	
	var data = req.body.Persons;
	console.log(data);
	
	var models = await Person.createEach(data).fetch();
	if (models.length == 0) {
		return res.badRequest("No data imported.");
	}
	return res.ok(`${models.length} person(s) imported.`);
},
//...

Config a route to the `import_data` action

As usual, we need to define a route to the controller action in order to reach it. Please add the following line to config/routes.js.

'/person/import_data': 'PersonController.import_data',

Adding ejs file

Please create views/person/import_data.ejs file with the following content.

<form action="/person/import_data" method="post">
	<input type="file" accept="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" onchange="handleFile(this.files)" />
	<table class="table">
		<tbody id="tableBody">
		</tbody>
	</table>
	<input type="submit" />
</form>

<script src="https://unpkg.com/xlsx/dist/xlsx.full.min.js"></script>
<script>
function handleFile(files) {
	var table = document.getElementById('tableBody');
	table.innerHTML = "";
	Array.from(files).filter(file => file.type == "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet").forEach(file => {
		let reader = new FileReader();
		reader.onload = function(e) {
			var bytes = new Uint8Array(e.target.result);
			var workbook = XLSX.read(bytes, {type: 'array'});
			var ws = workbook.Sheets[workbook.SheetNames[0]];
			
			var data = XLSX.utils.sheet_to_json(ws);
			for (var i=0; i<data.length; i++) {
				table.innerHTML += `
					<tr>
						<td><input type="text" name="Persons[${i}][name]" value="${data[i].name}" /></td>
						<td><input type="number" name="Persons[${i}][age]" value="${data[i].age}" /></td>
					</tr>`;
			}
		}
		reader.readAsArrayBuffer(file);
	});
}
</script>

Try it out at http://localhost:1337/person/import_data.

Sending Email

Setting up Mailgun, a free email service

Mailgun is a powerful transactional Email APIs that enable you to send, receive, and track emails.

Register a Mailgun account

Please register a Mailgun account at https://www.mailgun.com/.

enter image description here

Optional: Add a Authorized Recipients

Without providing credit card information, we could only send email to authorized recipients. Please add an authorized recipient email address for testing.

enter image description here

Using Mailgun in Sails

Mailgun comes with a Javascript module. we may use it directly in our Sails app.

Install Mailgun module

As usual, we use npm install ... --save command to add this module to our project.

npm install mailgun-js --save

Add Mailgun account information to `config/custom.js`

Please add your Mailgun account information according to your dashboard. mailgunDomain is your Mailgun domain, for example sandboxef5f56337bca4a36a491e6a91e8cd296.mailgun.org. mailgunSecret is the API Key showing on the domain dashboard. mailgunFrom is the email address showing in From of an email.

mailgunDomain: 'yourmailgundomain',
mailgunSecret: 'yourmailgunkey',
mailgunFrom:'yourmailgunfrommail',

Generate a helper function

We would like to construct a helper function, which would enable us to use the function globally in Sails. Please use the following command to create a blank helper function with name send-single-email.

sails generate helper send-single-email

Adding code to helper

Please replace the codes in api/helpers/send-single-email.js as follows.

var mailgun = require('mailgun-js')({apiKey: sails.config.custom.mailgunSecret, domain: sails.config.custom.mailgunDomain});

module.exports = {
	friendlyName: 'Send single email',
	description: '',
	inputs: {
		options:{
			type:'json'
		}
	},
	exits: {
	},
	fn: async function (inputs, exits) {
		mailgun.messages().send(inputs.options, function (error, body) {
		
			if(error) return exits.error(error)
			return exits.success(body);
		});
	}
};

Using the helper function

We could add the following to config/bootstrap.js for testing. Each time we sails lift, there is an email sending to destination@email.com. Please modify destination@email.com to the authorized email you have just added.

await sails.helpers.sendSingleEmail({
	to: 'destination@email.com',
	from: sails.config.custom.mailgunFrom,
	subject: 'Subject',
	text: 'Your message'
});

After that, just run sails lift to see result.

Example: Render ejs file as email content

We could use ejs rendering result as email content. sails.renderView(pathToView, templateData) function could render the ejs file content with dynamic data, and return the result as string. We could use option layout: false to exclude layout.ejs. Please add the following code to api/controllers/UserController.js.

userlistemail: async function(req, res) {
	var models = await User.find();
	var html = await sails.reanderView('user/email_userlist', {
		users: models,
		layout: false
	});
	await sails.helpers.sendSingleEmail({
		to: 'change_to_your@email.com',
		from: sails.config.custom.mailgunFrom,
		subject: 'Subject',
		html: html
	});
	return res.ok('users list sent!');
},

Please create views/user/email_userlist.ejs with the following content.

<% users.forEach(function(user) { %>
<div><%= user.username %></div>
<% }); %>

Lab 2 - Handling file upload and sending email

HTTP File upload & download

File upload

File Upload Example 1: Upload file using HTMLFormElement to server file system

Adding avatarPath attribute in api/models/User.js.

Add a upload action in api/controllers/UserController.js

Config a route to the upload action

Adding ejs file

File Upload Example 2: Upload file data to Database with <input type="hidden"> in <form> element

Adding avatar attribute in api/models/User.js.

Adding upload2 Controller action

Config a route to the upload2 action

Adding ejs file

File Upload Example 2.1: Convert file to Data URI and store in Database

Install datauri module

Changing upload action to store Data URI in UserController

Dynamic File download in Sails

File Download Example 1: Stream file content from server file system

Adding avatar Controller action

Adding route to controller action

Usage

File Download Example 2: Embed file content Data URI in HTML element

Adding a controller action

Adding the route

Adding view ejs file

Processing Excel file

SheetJS js-xlsx

Excel Example 1: Upload Excel file data to Database

Adding controller action

Adding ejs file of upload form

Adding route to the controller action

Excel Example 2: Export database as Excel file

Create a controller action

Setting route to the controller action

Excel Example 3: Parsing XLSX file data in Browser with client-side Javascript

Adding upload2 Controller action

Config a route to the import_data action

Adding ejs file

Sending Email

Setting up Mailgun, a free email service

Register a Mailgun account

Optional: Add a Authorized Recipients

Using Mailgun in Sails

Install Mailgun module

Add Mailgun account information to config/custom.js

Generate a helper function

Adding code to helper

Using the helper function

Example: Render ejs file as email content

Adding `avatarPath` attribute in `api/models/User.js`.

Add a `upload` action in `api/controllers/UserController.js`

File Upload Example 2: Upload file data to Database with `<input type="hidden">` in `<form>` element

Adding `avatar` attribute in `api/models/User.js`.

Adding `upload2` Controller action

Config a route to the `upload2` action

Install `datauri` module

Changing `upload` action to store Data URI in `UserController`

Adding `avatar` Controller action

Adding `upload2` Controller action

Config a route to the `import_data` action

Add Mailgun account information to `config/custom.js`