How to Save Data Locally in Electron

Steven NgCoding, Daily Debug BlogLeave a Comment

So you’ve hitched yourself on the Electron bandwagon, and you’ve started coding, and now you realize you want to save data locally in Electron. Yes, you know how to code and you can do whatever you want, but you obviously want to do the right thing for your application.

While I’ve been admiring Electron from afar for quite some time, I’ve finally dipped my toe into the Electron waters and encountered the same dilemma. After plenty of Binging and Googling, I found a lot of Stack Overflow pages and blog entries that pointed out a lot of popular libraries and NPMs to use, but less so on how to choose the best approach for my use case.

Save Data Locally in Electron

Save data locally in Electron

This article is about how to save data locally in Electron. So I’m not going to talk about solutions for client-server scenarios where you want to save data to a database server or to the cloud. This article is also about application data as opposed to content. By application data, I’m talking about the consolidated data that you expect only your application to read and write from, even though that may include user generated content. I’m not talking about self-contained “content” files like Word, Excel or JPEG files.

Also, let me get this out of the way now, I’m not going to prescribe any particular library or technique in this entry. My goal here is to help you figure out what your requirements are and move on from there.

There is no panacea with respect to how you save Electron data. Just do what is best for your particular application and your programming style and capabilities. What works for someone else’s application might not work for you.

In the end, you need to make a choice that’s effective for your application now, and ideally one that can grow with your application.

First Things First

What are you trying to accomplish? You’re probably inclined to say “I’m trying to save data locally in Electron, you fool!”

Ok, fair enough. But let’s figure some things out first. Let’s ask ourselves some pointed questions:

  • How much data do you think your application will generate in the short term? What about the long term?
  • How complicated is your data? Do you have a lot of nested relationships?
  • Do you want the datafile to be easily manipulated by users outside of the application?
  • Does the data need to be human readable?
  • Will the data need to be portable?
  • Does the data need to be readable by other applications while your Electron application is running? 1

I’m not going to answer each question directly, but you should always keep these questions in the back of your mind when evaluating the appropriateness of a save solution for your Electron application.

Two Key Decisions

In my mind, there are two decision points that you need to consider before choosing a final approach.

Electron’s Built In Storage or File Based Storage

Electron’s basically a browser on steroids that can be packaged up as an application. Because Electron has Chromium (Chrome) at its heart, you have most of the common HTML5 storage options at your disposal. That includes cookies, LocalStorage and IndexedDB. The catch, however, is that this data is stored in the user data folder and it’s not easily viewed or manipulated with your code editor. Because of that the data is also not portable in the same way a normal data file is. Depending on your application’s use case, this may be perfectly fine.

There are also some implications related to backing up the data. Time Machine or Windows File History can back the Electron data up, but it will seem like a black box to your users, and depending on who your users are, this can be a potential red flag.

If Electron’s built in storage isn’t suitable for your use case, however, then you want to look at file based options, such as a flat file like JSON, or an embedded database like SQLite.

Asynchronous or Synchronous

If you’re a dyed-in-the-wool Javascript fiend, this is an easy choice. Unless you have a specific reason not to, you’re going to go with an asynchronous solution.

If you’re coming from another language like Ruby, however, you may want the same degree of I/O predictability provided by the language you’re already used to. While the benefit of asynchronous data stores is that they’re non-blocking, sometimes blocking is a good thing, as you may not want to be forced to use callbacks or promises to avoid out of order read/write operations. If you’re not accustomed to using callbacks and promises, asynchronous I/O has the potential to destroy your sanity.

While most saving APIs for Electron are asynchronous, there are some options for those who prefer synchronous APIs.

Electron’s Built-In Storage

The methods available out of the box with Electron are tried and true, and a known quantity. Here’s what you have at your disposal:

  • Cookies
  • Session Storage
  • Local Storage
  • Web SQL
  • Indexed DB

Cookies, Session Storage and Local Storage

Cookes, session storage and local storage aren’t very useful for saving large amounts of data, but they are great for storing settings and application state. They are, however, not great for handling complex application data. If your application settings don’t need to be portable, then you will probably want to forgo these options.

Web SQL

WebSQL is basically SQLite. Not horrible in itself, but it’s been dead-ended as an HTML5 standard since 2010. You never know when this feature will be dropped from Chromium/Electron, so you probably don’t want to use this to save data locally in Electron. If you like SQLite, then the answer is, to simply use SQLite.

IndexedDB

IndexedDB is similar to NoSQL databases like Mongo or CouchDb. The structure revolves around collections of JSON objects. IndexedDB is a solid choice for saving your application data, but be aware that to access that data outside of Electron, you’ll have to provide some export functionality to your users.

File Based Storage – Text Files

Text files are very flexible. If your application isn’t going to produce a ton of data, then a text file might be a good way to go.

In most cases, text files are in plain text and are human readable. This means they are not appropriate for storing secrets, and they’re subject to manipulation by outside applications.

While easy to work with, text files are not without their share of tradeoffs. Text files can also be inefficient to load, as you may encounter heavier CPU and memory workloads as the file size increases.

The bottom line– if you know your application is going to have a lot of data, go with an embedded database instead.

JSON

If you’re using Electron, you know Javascript and you know JSON. Node works with JSON files natively, so JSON is a no-brainer right? Nope, JSON is a brainer.

Yes, Node handles JSON files quite nicely… provided they’re not too large. If your application has a lot of data (i.e., tens to hundreds of megabytes), you’ll want to switch to an embedded database.

JSON is human readable, and easily modified outside of your application – that is both good and bad. You obviously won’t want to store secrets like passwords in a plain text file like JSON.

JSON is also super easy to read and write. I already use a library called fs-extra, and it extends the Node fs library with sync and async commands for reading and writing JSON files without having to do any stringifying or parsing.

So in a nutshell, if the data volume of your app is modest, JSON is a very easy and efficient way to save your data.

Everything Else

You can save your data in flat text files in any format you want, whether it be XML, CSON, YAML or something else.

Unlike JSON, however, you’re going to have to convert the data to and from Javascript Objects every time you load or save. Depending on the complexity of the parsing, it can be CPU intensive and slow things down noticeably. Unless you have a specific goal of interoperability with other applications, you might want to forego this option.

File Based Storage – Embedded Databases

SQLite

If you are comfortable with relational databases, SQLite is a solid option. You get a file based API that gives you plenty of room to grow. You can use other tools to read SQLite files, such as a database query tool. There are a bunch of libraries that let you use SQLite, but you might want to give better-sqlite3 (https://github.com/JoshuaWise/better-sqlite3) a spin, as it advertises itself to be more performant than the other SQLite libraries.

Worth mentioning is that SQLite can handle a JSON data type, which does make life easier in terms of handling the data.

Since relational databases have schemas, you need to think about interoperability with older versions of your data files as you update your application and database schema. You will need to use a migration library to upgrade older database schemas to new ones. While this can be handled without too much headache by using with a preexisting library, it is something to think about. You may be able to forego the migrations if you use an ORM (object relational mapper) or a NoSQL wrapper that can write to SQLite.

NeDB

NeDB (https://github.com/louischatriot/nedb) is an embedded database that has an API similar to MongoDB. NeDB allows you to use a local file in the same fashion as SQLite.

It’s not that hard to use, and while not as fast as a real MongoDB database, it’s fast enough to be used with Electron. NeDB can be used to write to a file, or to one of Electron’s built in data stores. To be sure that it writes to a file, you do need to make sure you initialize it with a file path.

I’m not going to go into any further detail about NeDB for one simple reason– I haven’t had the need or opportunity to use it yet.

A Note on…

Storing Secrets

If your application is storing any secret data like passwords and API keys, you may not want to store those using any of the methods explained above. There is a library called Keytar (https://github.com/atom/node-keytar), which works well for Windows and Mac users. It can read and write data to the Windows Credential Manager or Mac Keychain, respectively. It will work on Linux, but it’s not as seamless as it is for Windows and Mac.

You can use Keytar in combination with any of the storage methods discussed in this blog by simply storing the ID of the secret in your file (as opposed to the secret itself).

API Wrappers

There are some wrappers that can use SQLite or IndexedDB as a back end while providing a different API. I have no real opinions on them, but if they happen to have an API that fits how you like to program, they can be a huge time saver.

For one app I wrote, I used Dexie.js (http://dexie.org/) to handle storing data to Electron’s IndexedDB. Some other options include Keyv (https://github.com/lukechilds/keyv) and PouchDB (https://pouchdb.com/).

If you’re using SQLite, you might want to use an ORM (object relational mapper) like Sequelize (https://sequelize.readthedocs.io/en/v3/) or bookshelf.js (http://bookshelfjs.org) to avoid having to write SQL. Whether you should use an ORM really depends on how you like to do things. ORMs do add weight to your application though, and frankly speaking, SQL is not a hard language to learn.

If there’s an API that mimics the way you’ve done thing in the past, go with that. If you don’t mind learning the direct API, then do that. Do you enjoy rolling your own wrapper?…do that instead. The best and only advice I can give is to take the approach that gets you forward progress the fastest.

Ok, Let’s Recap

The best options for saving local data, in my humble opinion, are: IndexedDB, JSON files, SQLite and NeDB.

Electron Built-in Storage: IndexedDB

The Good Parts

  • Storing reasonably large amounts of data
  • Storing complex data
  • Many wrappers are available to give you a choice in API

Hit Or Miss

  • Asynchronous API
  • Data can’t be read by other processes while Electron is running

The Not So Good Parts

  • Data portability
  • Easy access by end users outside of Electron

File-Based Storage: Text Files – JSON

The Good Parts

  • Storing modest amounts of data
  • Storing complex data
  • Data Portability
  • Human Readable
  • Super easy to implement
  • Easily Backed Up
  • Easy – no API wrapper required
  • Because you’re just saving a Javascript Object, you can organize it however you want

Hit Or Miss

  • Data can be manipulated by anyone with a text editor
  • Data can be manipulated by other applications while Electron is running
  • Can be done synchronously 2

The Not So Good Parts

  • Storing large amounts of data
  • Plaintext storage — there is no obfuscation out of the box

File-Based Storage: Embedded Database – SQLite

The Good Parts

  • Storing reasonably large amounts of data
  • Storing complex data
  • Data Portability
  • Not plain text
  • Easily Backed Up
  • Not hard to implement
  • People who are familiar to RDBMS
  • ACID compliant

Hit Or Miss

  • Synchronous API available 3
  • Object Relational Mappers (ORMs) can make coding easier
  • Because SQLite is a standard, files can be read by other applications
  • You should plan your schema in advance

The Not So Good Parts

  • When not using an ORM, requires knowledge of SQL, which can scare some people away
  • Requires knowledge of how to create a proper relational database
  • Schema changes over time need to be managed and handled automatically 4

File-Based Storage: Embedded Database – NeDB

Good Parts

  • Schemaless data
  • Storing large amounts of data
  • Storing complex data
  • Data portablity
  • Not plain text
  • MongoDB-like API

Hit Or Miss

  • API is Asynchronous
  • NeDB files aren’t as easily read by other applications as SQLite
  • Database cannot be accessed by other applications while open by Electron 5
  • NeDB will save to Electron’s IndexedDB if you don’t provide a filename

Not So Good Parts

  • API might feel foreign to people who are more familiar with RDBMS
  • Might not provide ACID transactions 6

Some Final Words

First thing – don’t take my word for anything. I am literally just another guy messing with Electron. Look around, do your own research. The more informed you are, the better decision you’ll make for your application.

The biggest decision you will need to make is whether to use Electron’s built-in options or a file-based option. There are pros and cons to either approach. You can always change your mind in the future, but depending on the volume and complexity of your application data, it can be a nuisance to migrate later.

Having said that, don’t overthink or overengineer how you save your application data. Put your application’s requirements first. If you’re well funded and you are writing an application with a clear long term roadmap, then yeah, you can afford to overthink things. If you’re a bootstrapped startup and throwing a spaghetti app out there hoping that it will stick, go with the fastest and easiest method for you to ship. You don’t know that the app will be successful, and it can’t be successful if it’s not out there. Balance your resources (i.e., time and money) with your release schedule in making your decision.

Now get out there, build your baby and ship it.

Footnotes

  1. This is a non-issue for most people, but there are some use cases where it might be a dealbreaker. For example, if your Electron app is creating job definitions for a scheduled task that is being handled by a separate executable.
  2. I use fs-extra (https://github.com/jprichardson/node-fs-extra) to read/write the file
  3. Provided you use better-sqlite3 (https://github.com/JoshuaWise/better-sqlite3) Because better-sqlite3 is synchronous, it probably won’t work with many ORMs.
  4. The easiest way to do this is with a database migrations library. Some ORM libraries include this feature.
  5. https://github.com/louischatriot/nedb/pull/535
  6. https://github.com/louischatriot/nedb/issues/398

Extrata - Save data locally in Electron

 

Assign It To Us uses Electron to develop Extrata.info

 

Quick Hit: Everyone Wants To Go Home During Extra Innings — Maybe Even The Umps

Steven NgAnalyticsLeave a Comment

FiveThirtyEight has a fascinating article on the tendencies of umpires in extra innings when it comes to calling strikes and balls:

Altogether, teams that are in a position to win get up to a 27 percentage point increase in the rate of called balls, while teams that look like they’re about to lose see increased strike rates of up to 33 percentage points.

I can’t imagine the umpires or the umpire’s union is going to be very pleased with this report.

Last month, FiveThirtyEight had another interesting article about which NBA teams are wronged by the refs the most, and the NBA referee’s union apparently called it “fake news”, in spite of the fact that the NBA provides the last two minute reports used by the article’s analysis for everyone to see.

Quick Hit: Create Visualizations Using YAML/JSON files

Steven NgAnalytics, Business Intelligence, VisualizationLeave a Comment

Daniel Kantor has a pretty nifty repo called just-dashboard that lets you make charts using YAML (Yet Another Markup Language) or JSON (JavaScript Object Notation).

YAML and JSON for the most part are human readable and don’t have a huge learning curve to pick up. One neat feature is that the library will work with Github Gists. You can see the documentation with samples here.

In case you’re wondering, it looks like the main visualization library just-dashboard relies on is D3.js.

Quick Hit: Craig Kerstiens’s Postgresql Hidden Gems

Steven NgDaily Debug Blog, DatabasesLeave a Comment

Postgresql guru Craig Kerstiens has a long but great list of Postgresql hidden gems submitted by his Twitter followers.

I’m one of those guys who never finds time to pore over documentation and only look up commands when I need them; so I found this particular psql tip to be very useful:

@jbrancha – In psql, setting ‘\x auto’ so that wide table results get displayed vertically!

Don’t like that ‘s in the title? Relax

 

How to Send Reports to a Slack Channel

Vince IarusciKnodeo, Pentaho, ReportingLeave a Comment

My work colleague Steven, often tells me that it bothers him when he’s forced to visit and log into many places to get the company reports he needs. He wants to avoid logging into separate reporting portals, intranet sites, file shares or native apps and would rather have his info prepared specifically for him on the communication tools he currently uses like email or Slack.  I can relate, because the team constantly uses the Slack tool to share ideas, reports, updates, news, notifications, system status, backup logs etc. In addition, this info is all in one place for all to comment on and it’s the one app that’s constantly open on our screens. So it made sense that we should send reports to a Slack channel.

Yes, Slack is a popular messaging app used by work teams. It also comes with a lot of services and an API for developers to integrate with their applications. In today’s post, we’ll use the files.upload method and a simple CURL command to automatically send a daily data warehouse report to a Slack channel.

Click on this link to download all the components for this post –>  Slack_Reporting_Example.  An additional download link is also included at the end of this article.

This image shows how an automated job runs a report against our data warehouse and posts the “My Daily Report” to the #knodeo Slack channel at a scheduled time every morning. The example below shows the same report posted as an image or PDF attachment.

Slack page with Reports

Steps on how to send reports to a Slack channel

Here are the basic steps to send reports to a Slack channel:

  1. Create the My_Daily_Report in Pentaho Report Designer
  2. Create the My_Report_Slack Transformation in Pentaho Data Integration
  3. Automate and schedule the Transformation to run each morning

1. Create My_Daily_Report in Pentaho Report Designer

If you’re going to post your daily performance indicators or metrics to your Slack channel, you’re going to need a report.  For our purposes, we use the open source Pentaho Report Designer to create a simple report.  The report is based on sample data from the Sample-Superstore.xls data table using the Orders tab.  The chart uses the Category field for the chart series and Sales for the values. The report is saved as My_Daily_Report.prpt which will be called upon in the transformation.

Pentaho Report Designer Slack Report

2. Create My_Report_Slack Transformation in Pentaho Data Integration

Create the My_Report_Slack transformation in the Pentaho Data Integration tool. The transformation does 2 things:

  1. Runs the report and saves the output to a file share
  2. Runs the CURL command to post the message with the report attachment to the Slack channel

PDI Transformation for Slack Post

Transformation Steps

2.1. Define Report Paths (Add Constant Rows)

The Add Constant Rows step sets the source report and the output file. The metadata fields defined are Report Name, Report Defn, and Output File. Adding additional rows & parameters will let you run multiple reports. In our example, we set the Report Definition to the saved report that we created in the Report Designer.

Set Slack Report paths

2.2 Run the Pentaho Report (Pentaho Reporting file output)

The next step in the flow uses the Pentaho Reporting file output step. It picks up the Report definition file and the Output file settings from the previous step. In addition, we can define the output format in the Output Processor section.

Run Pentaho Report for Slack

2.3. Define the Slack Message batch file to run (Get File Names)

Use the Get File Names step to select the batch file that contains the CURL command to post the report output to the Slack channel.  Within the code, the files.upload url will post the message with the file attachment. Here’s the code in the Slack_Message.bat file…



cd C:\Users\Demo User\Documents\Slack_Reporting_Example

curl -F file=@My_Daily_Report.pdf -F channels=#knodeo -F filename="AITM Reports" -F title="This is your Daily Report" -F token=xxxx-xxxxxxxxx-xxxxxx https://slack.com/api/files.upload

Replace xxxx-xxxxxxxxx-xxxxxx with your Slack API token.

Slack CURL command to post report

2.4. Execute the Slack batch file (Execute a Process)

The Execute a Process step will run the Slack_Message.bat from the previous step.

Run Slack command to post report

3. Automate the transformation to run daily.

We have our reports created and the Slack post process working. Now we need to schedule the transformation to run every morning so that Steven gets his daily report in the reports channel. That’s done using the Pentaho Pan command that runs the transformation. The command runs the transformation from the AITU_BI_F repo and creates a log file. To automate delivery, create a task in Microsoft Scheduler with a schedule to run daily at 7:00am.

Here’s the command that runs the transformation…



cd "/Apps/pdi-ce-7.1.0.0-12/data-integration"

pan.bat /rep:AITU_BI_F /file:"C:\Users\Demo User\Documents\Slack_Reporting_Example\My_Report_Slack.ktr" /level:Basic > "C:\Users\Demo User\Documents\Slack_Reporting_Example\My_Report_Slack.log"

Download the files for the Slack_Reporting_Example

(All steps of this example use open source tools.  These tools are part of the Knodeo Open Source Business Intelligence toolbox)