9 Methods to Retailer Information With a Raspberry Pi (how one can use included) – Chip Wired

Storing knowledge is the following step after accumulating it. This information will take you thru how a Raspberry Pi can retailer knowledge it has collected; knowledge that's for use for later evaluation. Storage is a crucial a part of the information evaluation course of (assortment, storage, evaluation, motion), and that is a part of the collection of guides I’m writing on that course of.

A Raspberry Pi can retailer knowledge in an Excel file, a CSV file, in a database, and over the web utilizing its WiFi or Ethernet connections. One of the simplest ways to retailer small quantities of knowledge, akin to that collected by a sensor linked to the Raspberry Pi, is through the use of a CSV file, an SQLite database, or TinyDB. These knowledge storage strategies are appropriate with the packages of Python which can be obtainable for Raspberry Pi.

I’ve discovered the perfect methods to get knowledge out of the Raspberry Pi are by:

  • Importing that knowledge to the web, akin to through the use of cloud storage or an HTTP API
  • Saving to a USB flash drive after which copying to a different pc
  • Utilizing Community Hooked up Storage that the Raspberry Pi can entry through Ethernet or WiFi
Raspberry Pi knowledge storage strategies

I’ve additionally discovered it’s doable to avoid wasting small quantities of knowledge utilizing the serial connection obtainable through the Raspberry Pi’s GPIO, although that is sometimes solely appropriate for small quantities of knowledge.

On this information I’ve checked out:

  • Totally different file codecs obtainable for Raspberry Pi (e.g. Excel recordsdata, and CSV recordsdata)
  • Databases {that a} Raspberry Pi can use (e.g. SQLite, and TinyDB)
  • The way to retailer Raspberry Pi knowledge away from the system itself (cloud storage, HTTP API, USB flash drive and so forth.)

1. Excel file

A Raspberry Pi can retailer knowledge in an Excel file utilizing LibreOffice Calc or by coding an app in Python. Information might be saved within the .xlsx format, which may then be opened by Excel. Some superior options of Excel recordsdata, akin to customized formatting and complex formulation, could not work when saved utilizing a Raspberry Pi.

Having the Raspberry Pi retailer knowledge in an Excel format might be nice for:

  • Storing knowledge that’s going to be analysed by Microsoft instruments later (e.g. PowerBI, Excel)
  • Making ready graphs, charts, or pivot tables primarily based on knowledge captured by a Raspberry Pi
  • Formatting knowledge for show

I’ve discovered Excel to be a helpful format for storing knowledge that might be later analysed utilizing Microsoft instruments, significantly Excel and PowerBI. This knowledge might be captured from sensors utilizing a Raspberry Pi, saved in an Excel file, after which analysed by these instruments on a bigger / extra highly effective pc. Should you’re serious about how the Raspberry Pi can accumulate knowledge, I wrote a separate information to that right here: chipwired.com/collect-data-with-raspberry-pi/

To have the Raspberry Pi retailer knowledge in an Excel file:

  1. Use Python pip to put in the openpyxl module
  2. Write an app to gather knowledge
  3. Use openpyxl to put in writing your knowledge to the workbook
  4. Save the file as .xlsx

I wrote an entire information on the other ways a Raspberry Pi can use .xlsx recordsdata (together with Python and openpyxl), test it out right here for extra element: chipwired.com/r-pi-open-excel-files/

I ran openpyxl on my Raspberry Pi to switch Excel sheets

Some disadvantages of utilizing Excel recordsdata for storage are:

  • Microsoft Workplace doesn’t run natively on Raspberry Pi. This implies the complete vary of Microsoft Excel options won't ever be obtainable on a Raspberry Pi making it more durable to format knowledge and analyse utilizing pivot tables and charts
  • It’s more durable to learn and write Excel recordsdata in different programming languages – if you wish to share this knowledge with an app written in a unique language (akin to OCaml), you is likely to be higher off storing in a unique language
  • Excel recordsdata created by a Raspberry Pi could not work effectively with different apps, even these which can be designed to learn Excel recordsdata

2. Cloud storage

Information collected by a Raspberry Pi, together with knowledge from sensors, might be saved within the cloud. The Raspberry Pi can entry standard cloud storage companies akin to Amazon AWS and Microsoft Azure, and it will possibly additionally talk with bespoke cloud companies akin to JSON Bin. Sensor knowledge might be saved within the cloud by having the Raspberry Pi use the related cloud storage API.

Some advantages of utilizing cloud storage with a Raspberry Pi embody:

  • Information is accessible from wherever with an web assortment, it’s not restricted solely to the Raspberry Pi used to gather the information
  • Analysing the information can completed utilizing a extra highly effective processor, both utilizing cloud companies or by downloading the information to a extra highly effective pc
  • Information is backed up mechanically

I really feel that that is the longer term: Small low-powered computer systems which work together with folks and the world but do all their processing and knowledge storage on the cloud. We are able to create a community of IoT sensors utilizing Raspberry Pi, or change our desktops with Raspberry Pi, and in each instances utilise the cloud to perform the duties we wish to do. Utilising cloud storage and companies is an ideal match with Raspberry Pi.

To make use of cloud storage with Raspberry Pi:

  1. Join an account with a cloud storage supplier
  2. Setup the Raspberry Pi to speak with the cloud utilizing their most well-liked technique (that is often through an HTTP API)
  3. Acquire knowledge on the Raspberry Pi
  4. Connect with the cloud and add the information

Free cloud storage suppliers I like to make use of embody Pantry Cloud and JSON Bin. I additionally like Linode as a cloud storage supplier, although they don't have a free choice obtainable (it solely begins at $5/month although!). The large names in cloud storage are Azure by Microsoft, Amazon AWS, and Google Cloud.

Drawbacks of utilizing cloud storage on Raspberry Pi embody:

  • An web connection is required. This will increase the quantity of energy the Raspberry Pi makes use of and can also restrict the vary on location the Raspberry Pi must be situated in (it will possibly’t be too removed from a WiFi entry level)
  • An account with a cloud storage supplier is required. This could improve the value of a undertaking as free choices are often restricted to trials or small quantities of knowledge.
  • Cloud storage suppliers can have sophisticated APIs or steps required to hook up with their cloud; it’s not so simple as storing to a file situated on a Raspberry Pi.

3. CSV file

CSV (Comma Separated Values) recordsdata are one of many best methods to retailer knowledge collected by a Raspberry Pi. A Raspberry Pi can retailer knowledge in a CSV file utilizing Python or any programming language that may run on a Raspberry Pi. CSV is an open format for file storage that's obtainable on Raspberry Pi.

Use CSV with a Raspberry Pi when:

  • Saving easy knowledge that might be analysed by different apps or programming languages
  • That you must save bandwidth or cupboard space (CSV is a really compact and environment friendly file format)
  • Must shortly learn the information your self, together with if you want to modify small quantities of knowledge manually

CSV recordsdata can simply be learn and opened utilizing textual content processing apps, in addition to spreadsheet instruments. Most of the instruments I point out in my information to Excel recordsdata additionally work effectively with CSV recordsdata (take a look at that information right here: chipwired.com/r-pi-open-excel-files/).

I’ve discovered that CSV is a good format to make use of for saving small quantities of knowledge from a Raspberry Pi, knowledge that might be analysed later. I’ve used CSV to file temperature readings over time, counts of a movement sensor being activated, and samples of a voltage stage studying.

To make use of CSV on a Raspberry Pi:

  1. Add the CSV library or module to whichever programming language you’re utilizing (in Python it’s referred to as csv)
  2. Open a CSV file
  3. Append a sensor studying or different knowledge to the file
  4. Shut and save the file

The CSV file can then be opened by an app to be used and evaluation, both on the Raspberry Pi or on one other pc. CSV recordsdata may also be uploaded to the cloud for evaluation.

Excel recordsdata might be transformed to CSV (take a look at my information here for particulars)

Some downsides to CSV recordsdata embody:

  • Incapacity to avoid wasting any evaluation within the file. CSV is a format for saving knowledge solely, any evaluation, graphs, pivot tables, or different info have to be saved in a unique file
  • No strategy to format, summarise, or annotate the information
  • Some apps could disagree on the character used to separate rows and columns in knowledge – this implies having to grasp separators which I discover annoying given the target of simplicity that CSV is chasing

4. SQL database

A Raspberry Pi can talk with an SQL database to retailer knowledge. This consists of creating new tables, storing knowledge within the tables, and even querying that knowledge to carry out evaluation. A Raspberry Pi can host an SQL databases on itself, or it will possibly join to at least one remotely.

Storing knowledge in an SQL database is the everyday remedy of any knowledge collected, together with that collected by a Raspberry Pi. Working with SQL databases is among the first issues I learnt when studying about knowledge evaluation.

SQL databases are nice for:

  • Dependable recording and long-term storage of knowledge (the database will usually guarantee every file is error free, and that the storage doesn't comprise errors)
  • Storing knowledge that matches a “desk” construction, together with knowledge “transactions” (akin to frequent temperature readings)
  • Simple strategy to ‘append’ knowledge to an information set (in SQL, it's a single assertion so as to add an information studying; in different strategies akin to CSV, it will possibly contain studying all the knowledge set earlier than appending a single knowledge studying)

To make use of an SQL database on a Raspberry Pi:

  1. Setup an SQL database with a desk prepared to avoid wasting the information your Raspberry Pi sends
  2. Join your app to the SQL database (should you wrote the app your self, you’ll probably want to make use of the suitable library, module, or driver for the database that you simply wish to hook up with – e.g. sqlite3 in Python for connecting to an SQLite database)
  3. Save knowledge to the database utilizing the features within the library, module, or driver

A Raspberry Pi can host a database itself, or join to a different one. Should you’re serious about working a database in your Raspberry Pi (it’s not has arduous because it appears), take a look at my information to SQL databases on Raspberry Pi right here: chipwired.com/sql-databases-for-raspberry-pi/

SQLite working on my Raspberry Pi

A couple of downsides to utilizing an SQL database along with your Raspberry Pi:

  • A primary understanding of the SQL language will probably be required to retailer and later extract knowledge from the database
  • It may possibly really feel sophisticated to hook up with an SQL database from an app (it’s not as arduous because it appears)
  • It may possibly additionally really feel prefer it’s more durable to visualise knowledge in an SQL database (this turns into simpler as you be taught extra about databases)
  • A connection to a database is required. If that is hosted remotely, it's going to imply utilizing the Ethernet or WiFi on the Raspberry Pi probably costing extra energy to make use of.

5. NoSQL database

Databases that don't use SQL may even work on a Raspberry Pi. A Raspberry Pi can hook up with NoSQL databases, or it will possibly host them itself. NoSQL databases akin to MongoDB or TinyDB can run on a Raspberry Pi itself. A Raspberry Pi can even hook up with different NoSQL databases akin to Azure Cosmos DB or Cassandra.

I discovered an amazing variety of NoSQL databases that Raspberry Pi can hook up with. It’s been a bit more durable to setup a NoSQL database on a Raspberry Pi, which is why I didn’t embody many examples above. “NoSQL” is a little bit of a catch-all time period for databases that don’t observe the relational-SQL mannequin, so the kind and nature of every databases is sort of various.

NoSQL databases on Raspberry Pi are good for:

  • Easy knowledge storage, usually the interface to retailer knowledge is easier than even an SQL database
  • Should you don’t know what the general mannequin goes to be but. It's usually simple to alter the construction of a NoSQL database as your undertaking evolves; that is more durable to do with an SQL database
  • If the information evaluation might be achieved on one other pc. It won't be as simple to analyse knowledge saved in a NoSQL database working on a Raspberry Pi, it is likely to be higher to analyse this knowledge on one other pc

To make use of a NoSQL database on Raspberry Pi:

  1. Setup the NoSQL database on the Raspberry Pi (or establish which one you wish to hook up with)
  2. Set up the library or module for connecting to that database
  3. Write an app that collects knowledge, connects to the database, and shops that knowledge within the database

The simplest NoSQL database I discovered to work with on Raspberry Pi is TinyDB. I included it in my information to Raspberry Pi databases, which might be discovered right here (extra detailed directions included): chipwired.com/databases-for-raspberry-pi/

A NoSQL database (TinyDB) working on my Raspberry Pi

Some downsides to utilizing a NoSQL database embody:

  • Evaluation of the information generally is a little more durable when you have important quantities of knowledge; it might must be reworked or learn into one other system earlier than being analysed
  • Show and formatting of knowledge could also be equally more durable
  • A connection to a database is required – if the database is hosted remotely, this implies the Raspberry Pi should eat extra energy to take care of a WiFi or Ethernet connection to retailer knowledge within the database.

6. USB flash drive

A Raspberry Pi can use its USB ports to retailer knowledge on a USB flash drive. Raspberry Pi helps all kinds of flash drive {hardware} and file methods used to format flash drives. If utilizing a USB flash drive to retailer knowledge for a Raspberry Pi undertaking, it's endorsed to make use of a clean one particularly for this objective (i.e. keep away from sharing flash drives between Raspberry Pi initiatives and different makes use of).

Storing knowledge on a flash drive is nice whenever you want a Raspberry Pi to:

  • Seize a variety of knowledge, an excessive amount of to ship over the web connection obtainable
  • Retailer knowledge when a dependable web connection shouldn't be obtainable
  • Use much less energy by turning off the WiFi and Ethernet features

I discovered USB flash drives to be a great way to get recordsdata of saved knowledge out of a Raspberry Pi, significantly when there’s no web connection obtainable. The recordsdata mentioned earlier (Excel, CSV, and even SQLite databases) might be saved to a USB stick and simply transferred to a different pc.

To have a Raspberry Pi save knowledge to a USB flash drive:

  • Join the flash drive and format the flash drive to make use of a file format effectively supported on the Raspberry Pi (e.g. FAT32 or exFAT). Observe, formatting will delete knowledge already on the flash drive. A information to formatting might be discovered here.
  • Find the place the flash drive is mounted (from Terminal attempt this guide; from Python attempt this guide)
  • Use an app (akin to one you wrote) to gather knowledge and put it aside to the flash drive

A USB flash drive will not be a very good resolution if:

  • Python or Terminal work isn’t your power. I discovered it actually arduous to determine how one can mechanically get the Raspberry Pi to recognise the USB flash drive after which save knowledge to it (I needed to manually test issues every time I plugged within the flash drive)
  • That you must hold the scale of the Raspberry set up small (i.e. you possibly can’t have a flash drive protruding one finish)
  • Retrieving the USB flash drive might be troublesome, for instance if the Raspberry Pi undertaking is put in in a distant location, or someplace that’s out of attain for an individual to get to

7. Internet API

A Raspberry Pi can use net APIs as a strategy to retailer knowledge. For net APIs hosted on different servers, the Raspberry Pi can add its knowledge through the use of that APIs endpoints. The Raspberry Pi can even set up itself as an internet API server and obtain knowledge from different units to retailer in a database. Internet APIs are one of many strategies used to entry cloud storage.

When to make use of an internet API with Raspberry Pi:

  • Should you’re organising the Raspberry Pi as a server and it'll obtain knowledge from different units utilizing an internet API
  • If an web service you wish to use to retailer knowledge presents an internet API (e.g. a Tesla API)
  • The online API you’re utilizing presents knowledge evaluation functionality

I’ve discovered net APIs are a standard strategy to entry cloud storage (see above for particulars on how one can use cloud storage), nevertheless there are some net APIs that I wouldn’t name cloud storage. These might be particular to a selected use of Raspberry Pi, akin to connecting to a Tesla.

To join to an internet API utilizing a Raspberry Pi:

  1. Establish the API and browse its directions on how one can join
  2. Import the suitable module to hook up with the API based on the directions (sometimes for Python that is requests module)
  3. Observe the module’s directions to hook up with the API and trade knowledge

A Raspberry Pi may also be setup to serve an internet API. This may be achieved utilizing a framework akin to Flask.

Disadvantages of utilizing an internet API with Raspberry Pi are much like different internet-based storage strategies:

  • Must eat energy utilizing the WiFi or Ethernet connections
  • Might be troublesome to setup (relying in your Python coding abilities)
  • Show and formatting of knowledge could depend on utilizing one other device

8. Serial and GPIO

Information might be despatched to a storage system linked to the serial communication pins of the Raspberry Pi’s GPIO. As soon as one other system, akin to flash storage, is linked to the GPIO, an app working on the Raspberry Pi can ship knowledge to it. The Raspberry Pi helps SPI and I2C serial connections over the GPIO.

Utilizing the GPIO to retailer knowledge is helpful when you want to:

  • Retailer knowledge that might be utilized by one other microelectronics system
  • Retailer small quantities of knowledge that might be saved even when the Raspberry Pi loses energy
  • Use much less energy

For my part, utilizing the serial interface to retailer knowledge is greatest whenever you wish to preserve energy. It can use much less energy than establishing a WiFi or Ethernet connection each time you wish to avoid wasting knowledge, and it must also give an influence saving in comparison with writing to a USB flash drive.

I sometimes see flash storage vary from 1mb to 128mb (sure megabytes). The worth vary for this may be between $1 and $30 relying on the way you join it to the GPIO (should you can solder to a breakout board it’s cheaper than utilizing a breadboard).

To make use of the Raspberry Pi GPIO to retailer knowledge:

  1. Purchase flash storage that helps a serial (SPI or I2C) interface
  2. Plug the flash storage right into a breadboard, or solder it onto a breakout board that may be plugged right into a Raspberry Pi
  3. Import the suitable library or module (I discovered this tutorial useful) to your app
  4. Use your app to speak with the storage

Some disadvantages of utilizing the GPIO to retailer knowledge:

  • It’s sophisticated to setup. A complicated understanding of Python or C/C++ is required to speak with storage connected to the GPIO; some understanding of electronics (e.g. how one can join the flash reminiscence to the GPIO) can be wanted.
  • Small storage obtainable. Should you want greater than say 128mb of storage, it’s probably more cost effective to discover an alternate resolution.
  • It could improve the scale of your Raspberry Pi (as one other circuit must be constructed across the storage chip)

9. Community connected storage

A Raspberry Pi can use a NAS (community connected storage) to retailer knowledge, or it will possibly act as a NAS and produce other computer systems use it for storage. Connecting to a NAS includes mounting it as a part of the Raspberry Pi file system after which studying and writing knowledge to recordsdata saved on the NAS. Establishing the Raspberry Pi as a NAS includes putting in a SAMBA server and sharing storage linked to the Raspberry Pi.

I’ve discovered this extra helpful whenever you wish to setup the Raspberry Pi as a NAS to retailer knowledge from different computer systems on a community. The opposite strategies mentioned on this information are extra helpful if you wish to accumulate knowledge from a Raspberry Pi and retailer it elsewhere.

To attach the Raspberry Pi to a NAS:

  1. Guarantee the suitable communication protocol in your NAS is put in in your Raspberry Pi (that is sometimes smb)
  2. Mount the NAS in your Raspberry Pi’s file system
  3. Use recordsdata on the NAS as you'll any file regionally

A full information might be discovered here.

To setup the Raspberry Pi as a NAS, take a look at this information here.

Disadvantages for utilizing a NAS with Raspberry Pi are:

  • A community connection (Ethernet or WiFi) is required, this will use further energy
  • The community connection must be dependable, knowledge could also be corrupted if the connection is disturbed whereas accessing recordsdata
  • Storing the information could contain an additional step (e.g. writing to an Excel file, and saving that Excel file on the NAS)

Chris is constructing an entire information on how one can analyse knowledge with the Raspberry Pi. Information assortment, knowledge storage, and knowledge evaluation are the important thing pillars of this information.



Source link

Leave a Reply

Your email address will not be published. Required fields are marked *