COVID Hacks #3 Patient zero

Summary

COVID HACKS #4 - Patient Zero - the Chain Breaker

A novel solution to contact tracing and notifying authorities

The Problem

In these tough times, where the danger of community spread of the deadly virus haunts every city, public health officers are faced with the dilemma of tracing from where a particular patient contracted the virus or to whom he/she transferred it to. Many patients intentionally hide details and many are not able to recollect the people they met or places they went to. Also, while going to public places it is not always possible to maintain the required distance.

Solution

To solve the mentioned problems there are very limited number of devices deployed, mostly by private agencies but nothing at the personal level. I wish to make a module that can be mounted on a cap that warns the wearer if a person comes too close.

The module would also track the GPS location and click pictures of the persons the wearer has come in close proximity(less than 2m) to. The collected data would help trace down all possible people in danger if the wearer is found Covid-positive and timely quarantined.

Implementation

I plan to make a smart cap with the Arduino MKR 1010 WiFi as the main controller, fitted with a GPS module, TOF Proximity sensor and a camera module. The GPS module would help track a person's whereabouts and warn the wearer if he/she enters a hotspot.

The camera module captures images of people who come closer than 4m to the person on being triggered by the proximity sensor. If a person comes closer than 2m, a buzzer warns the wearer.

All imagery and GPS data is stored on a micro SD card. If the person is tested positive, the GPS data and the imagery data are analysed for identifying people at risk so that they may be quarantined in time.

---

 

Things used in this project

Hardware components
	Arduino MKR WiFi 1010	×	1
	DFRobot Husky Lens	×	1
	STMicroelectronics VL53L0X	×	1
	MikroE GPS Click	×	1
	Buzzer	×	1
Software apps and online services
	Arduino IDE
	Ubidots
Hand tools and fabrication machines
	Soldering iron (generic)
	Wire Cutter / Stripper, 5.25 " Overall Length

 

How it worked out

Circuit Diagram

Now I will show you the arrangement of various circuit components on the cap. I agree that the construction is pretty crude but the thing is that it does the job very well and is the most cost-effective way to build this if it were to be deployed on scale - which is a requirement for this contest.

Here, one can clearly see the camera lens facing the forward direction, with a Qwiic connector attached for power supply and I2C data lines for transferring image inference data to the Arduino board placed at the side.

Placement of Arduino board

The only component placed inside the cap is the power supply consisting in a 9V battery with a 7805 regulator hot glued firmly to the inside wall of the cap fabric

NOTE: In the above pic, the wires and the Arduino may be appearing to be floating in mid-air, but that is not the case actually as here I had to turn the cap for a better shooting angle.

The next component which is critical to the project is the microSD module which stores the facial recognition details of all the persons whom the wearer has come in close proximity to. Here, I have used a Samsung 8GB 10HC memory card but any card is fine, as speed is not much of a concern here.

The Piezoelectric passive Buzzer placed with the Arduino warns the wearer in case a person comes too near, i.e. less than 2m.

Since, I was not able to get a range of more than 1.8m with acceptable errors with a generic HC-SR04 ultrasonic sensor, I chose an Adafruit VL53L0X TOF LIDAR module for the range detection. As, I have explained earlier, if a person comes closer than 2m, the Husky Lens is woken up and the face of the person in the vicinity is learnt and stored in memory.

The final product looks really cool and futuristic.

Now that I have demonstrated the build I will explain how the Husky Lens recognises faces and stores them in memory.

---

 

The Husky Lens

Demosnstration of Husky Lens

Sleek...isn`t it

HuskyLens is an easy-to-use AI machine vision sensor with 6 built-in functions: face recognition, object tracking, object recognition, line following, color detection and tag detection.

Through the UART / I2C port, HuskyLens can connect to Arduino, Raspberry Pi, or micro:bit to help you make very creative projects without playing with complex algorithms.

Specification

• Processor: Kendryte K210
• Image Sensor:
  SEN0305 HuskyLens: OV2640 (2.0Megapixel Camera)
  SEN0336 HuskyLens PRO: OV5640 (5.0MegaPixel Camera)

NOTE : I have the 2MP version

• Supply Voltage: 3.3~5.0V
• Current Consumption(TYP): 320mA@3.3V, 230mA@5.0V (face recognition mode; 80% backlight brightness; fill light off)
• Connection Interface: UART; I2C
• Display: 2.0-inch IPS screen with 320*240 resolution
• Built-in Algorithms: Face Recognition, Object Tracking, Object Recognition, Line Tracking, Color Recognition, Tag Recognition
• Dimension: 52mm44.5mm / 2.051.75"

Board Overview

Buttons

There are two buttons on the HuskyLens, the function button and the learning button. The basic operations of these two buttons are shown as follows:

• Dial the "function button" to left or right to switch different functions.
• Short press the "Learning button" to learn the specified object; long press the "Learning button" to continuously learn the specified object from different angles and distances; if HuskyLens has learned the object before, short press the "Learn button" to make it forget.
• Long press the "function button" to enter into the second-level menu(parameter setting) in the current function. Dial left, right or short press the "function button" to set related parameters.

button usage

Face Recognition

This function can detect any face contour, recognize and track the learned face

The default setting is to learn and recognize a single face.

Operation and Setting

Dial the function button to the left until the word "Face recognition" is displayed at the top of the screen.

Learning and Detection

Face Detection:

Point the HuskyLens at any faces. When a face is detected, it will be automatically selected by a white frame with words "Face" on the screen.

Face Detection

Face Learning: Point the “+” symbol at a face, short press the "learning button" to learn the face. If the same face is detected by HuskyLens, a blue frame with words "Face: ID1" will be displayed on the screen，which indicates that HuskyLens has learned the face before and can recognize it now.

Face ID assignment

With this key feature, we can store more than 100 face IDs which can be extremely beneficial for contact tracing!!

And that is exactly what the goal of this project is!!

However, there is a problem if you have not already figured out - the pressing of the buttons.

An Arduino library takes this hassle away from the wearer and can be used to trigger the face recognition function internally through I2C signals on the occurrence of a particular event.

Setting up Husky Lens for the Arduino MKR1010 Wi-Fi

Please download and install the HUSKYLENS Library first.

• Unzip the file, then copy the folder to the "libraries" folder of the Arduino IDE. Then check whether the folder name is "HUSKYLENS". If not, please change it as "HUSKYLENS". Please note that the library file name must be HUSKYLENS.

• All .h files and .cpp files must in the root directory of the "HUSKYELSN" folder.

API Introduction

Please click the link below to view the API introduction documents.

https://github.com/HuskyLens/HUSKYLENSArduino/blob/master/HUSKYLENS%20Arduino%20API.md

Also, feel free to explore the example projects on the site to get a brief idea

Connecting to Ubidots Cloud over HTTP

OK, so we have been successfully able to raise an alarm if a person comes nearer than 2m from the wearer and to learn the person`s facial features and store it on a microSD card.

What next??

Lets connect it to the CLoud. The question that arises is which webservice should we use. There are so many out there with big players like Google, IBM and Microsoft and AWS and even our beloved Arduino.

However, I chose something completely off the block - Ubidots. These are steps that have to be followed to use Ubidots Cloud with our Arduino MKR1010 Wi-Fi Board.

1. Download the Ubidots library and click on

Sketch -> Include Library -> Add .ZIP Library.

Then, select the .zip file of Ubidots and then click on "Accept". Finally, close the Arduino IDE and open it again.

2.Sending (POST) Data to Ubidots

• After making sure your board is successfully connected with the computer, open the Arduino IDE and paste the sample code below. Once you have pasted the code, you will need to assign your Ubidots TOKEN, SSID (WiFi Name) and Password of the available network.

// This example sends data to multiple variables to
// Ubidots through HTTP protocol.

/****************************************
* Include Libraries
****************************************/

#include "Ubidots.h"

/****************************************
 * Define Instances and Constants
****************************************/

const char* UBIDOTS_TOKEN = "...";  // Put here your Ubidots TOKEN
const char* WIFI_SSID = "vardhan128";      // Put here your Wi-Fi SSID
const char* WIFI_PASS = "shiella21";      // Put here your Wi-Fi password

Ubidots ubidots(UBIDOTS_TOKEN, UBI_HTTP);

/****************************************
* Auxiliar Functions
****************************************/

// Put here your auxiliar functions

/****************************************
 * Main Functions
****************************************/

void setup() {
 Serial.begin(115200);
 ubidots.wifiConnect(WIFI_SSID, WIFI_PASS);
 // ubidots.setDebug(true);  // Uncomment this line for printing debug messages
}

void loop() {
 float value1 = random(0, 9) * 10;
  float value2 = random(0, 9) * 100;
 float value3 = random(0, 9) * 1000;
  ubidots.add("Variable_Name_One", value1);  // Change for your variable name
  ubidots.add("Variable_Name_Two", value2);
  ubidots.add("Variable_Name_Three", value3);

  bool bufferSent = false;
  bufferSent = ubidots.send();  // Will send data to a device label that matches the device Id

  if (bufferSent) {
    // Do something if values were sent properly
    Serial.println("Values sent by the device");
  }
  delay(5000);
}

• UPLOAD the code into your “Arduino MKR WIFI 1010”

To verify the device's connectivity and the server response, open the serial monitor to check if the data is being send correctly.

• To verify the device's connectivity and the server response, open the serial monitor by selecting the "magnifying glass" icon in the top right corner of the Arduino IDE to check if the data is being send correctly.

After you follow the above steps, you would be able to see a response in the serial monitor like the one below.

Finally in your Ubidots account you will be able to see the data posted in your device.

You will see that the device is automatically recognized by the platform.

Creating Dashboards for seeing data in UbiDots

Every Dashboard has certain elements with which it organizes information. Every widget has a different set of configuration options. Depending on the widget, you might be able to add one or several variables. For instance, the metric widget only supports one variable, while the line chart widget supports multiple variables.

Static Dashboards are used to visualize data from predetermined devices and variables. Static dashboards display the same device and variable data at all times.

• Static Dashboards are used to visualize data from predetermined devices and variables. Static dashboards display the same device and variable data at all times.

Dynamic Dashboards utilize a primary and replica (master/slave) technology to visualize data from different devices and variables, using a drop-down menu in the settings bar. By selecting different devices from the settings bar, a dynamic dashboard will refresh with the underlying device and variables populating each widget's visualization according to the device selected.

• Dynamic Dashboards utilize a primary and replica (master/slave) technology to visualize data from different devices and variables, using a drop-down menu in the settings bar. By selecting different devices from the settings bar, a dynamic dashboard will refresh with the underlying device and variables populating each widget's visualization according to the device selected.

Explaining Dynamic Dashboards

Explaining Dynamic Dashboards

When a device is selected from the setting bar, then all variables within each widget will automatically update to reflect the device’s variables. This is achieved because Dynamic Dashboard widgets are not associated with predetermined variables; instead, they are associated with variable labels (i.e., label = temperature).

This is a very helpful enablement as it erases the need to create additional dashboards per device. Imagine you have a fleet of 1,000 trucks; with static dashboards, you’d have to create 1,000 dashboards. With dynamic dashboards, you only have to create one dashboard, and then have the App's Users select the device for which they want to see the data:

Within the options to create a dynamic dashboard, you can choose which devices are going to be displayed in the drop-down list:

• All devices: The list of all the existing devices of the account.

• Device group: Only the devices that belong to the specific group selected.

• Device type: The devices that belong to the device type selected.

IMPORTANT NOTE: The map widget will display only the devices of the list.

Creating a Dashboard (static or dynamic)

Creating a Dashboard (static or dynamic)

To create a dynamic dashboard, go to "Data" –> "Dashboards"

Click on the "dashboard" icon in the top-left of the user interface:

Then click on the "+" icon:

Assign a NAME to your dashboard and provide some general settings:

• Default time range: The first load corresponds to the predetermined time interval when the dashboard is loaded, otherwise it corresponds to the last selected time range.

IMPORTANT NOTE: Customizing your Date format.

You can enter your own Date format using a parsing string as described in the Moment JS library. Below you can find quick parsing strings and examples:

Finally, click on the green icon in the lower right of your screen to save the configuration and create a new dashboard.

Dashboards interaction options

– Date time picker: allows selecting a time range to populate the dashboard with data contained within the range.

– Enable/Disable real-time: dashboards have the benefit of real-time update, meaning every time a new Dot comes in, it will be automatically reflected in the dashboard and its respective widgets. Sometimes it is useful to disable this feature to have a fixed time range and analyze that period without disturbing it with new data. To switch between real-time or not, just click the option accordingly.

Creating a Static Widget

• To create a static widget, click on the “+” icon in the top-right corner of the dashboard user interface.
• Select the type of Widget from the available options, or create your own with the HTML Canvas.
• Select Static Widget Assign a Variable for the widget by selecting "+ Add Variables"
• Configure the widget appearance and save

Creating a Dynamic Widget

• To create a static widget, click on the “+” icon in the top-right corner of the dashboard user interface.
• Select the type of Widget from the available options, or create your own with the HTML Canvas.
• Select Dynamic Widget
• Click on "+ Add Variables" to enter a Variable label
• Configure the widget appearance and save

Every widget has a different set of configuration options. Depending on the widget you might be able to add one or several variables. For instance, the metric widget only supports one variable, while the line widget supports multiple variables.

And with this we are done creating our dashboard on Ubidots!!

And with this we are done creating our dashboard on Ubidots!!

Now, we monitor the status of the wearer remotely and if he/she is declared COVID positive at some point, the facial recognition data can be extracted from the microSD card and people coming in proximity to that person can be warned in time.

Final Dashboard

Final Dashboard

My dashboard gives information about the body temperature of the wearer, the TOF sensor reading and the number of people the person has come in close proximity to in the last 14 days.

As you might have guessed all the entities in my dashboard are dynamic and update regularly at intervals of 10s which is better than the same integration with the Arduino IoT Cloud for which I have to give 40s update time at least for plausible readings.

The second field on my dashboard updates whenever the TOF sensor reading is less than 2m and facial capture is started by the Husky Lens.

A webhook is attached to the body temperature reading and in case it goes above 100F a warning is raised and an SMS is sent to the wearer to get a COVID test done.

This completes my project - Patient Zero which is sure to go a long way in helping humanity fight this pandemic and help Detect and Protect with Arduino

Future work and Improvements

The improvements that can be done are many -

- The Aesthetics can be improved with 3d-printed enclosures (I do not have access to my university lab right now, otherwise I could have easily done this)

- The VL53L0X LIDAR can be mounted on a servo motor for 360-degree coverage

- A LiPo battery can be used to reduce weight

- A better android app can be made

- Linking a NodeRed script through a webhook to the Aarogya Setu app which is the World`s largest open-source COVID status tracker made compulsory to be installed on every Indian`s smartphone. This would help me increase the scope for the usage of my product.

Conclusion

I must say that this project was one of the more successful of my projects and I was able to build something that can easily be used by people for personal safety. I know that it is not a marketable product but with some minor aesthetic improvements, it may as well be just that. Also, the integration with the Aarogya Setu app would make it a standard personal protection gear for a country as large as India which is a success in itself.

Thanks for reading!!

Watch out for updates!!

 

Update 1:

I have dropped the idea of using the Husky Lens to process the incoming imagery data as it can recognize only that much. Instead, I am using the Husky Lens just for image capture and sending imagery data over to a Raspberry Pi on which the inference engine built with the OpenVINO Toolkit runs.

From the model library, I directly used the human detection model that is pretty well reviewed. Initially, I did get some latency in image processing but I managed to solve that. The results were just amazing. Much accurate face detection and at multiple angles, something the Husky Lens could not achieve.

The overall architecture remains the same. However, with the implementation of the face detection inference engine built with the OpenVINO toolkit has increased the reliability of my solution to a great extent.

References

https://wiki.dfrobot.com/HUSKYLENS_V1.0_SKU_SEN0305_SEN0336#target_0

https://github.com/HuskyLens/HUSKYLENSUploader

https://ubidots.com/

https://forum.fritzing.org/t/part-request-arduino-mkr-wifi-1010/7617

https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit.html

...and any other source I have mistakenly not included

Schematics

Circuit Image

Code

• Publish all info to Ubidots
• Get Information from widgets on Ubidots portal
• Test publish command
• Husky Lens I2C trigger
• Husky Lens Face Detect on trigger
• Husky Lens Library

Publish all info to Ubidots

C/C++

/****************************************

 * Include Libraries

 ****************************************/

#include "UbidotsESPMQTT.h"

 

/****************************************

 * Define Constants

 ****************************************/

#define TOKEN "BBFF-HQzhjmDxosWtw94dhMISDcaLpZAzxK" // Your Ubidots TOKEN

#define WIFINAME "Moinak" //Your SSID

#define WIFIPASS "lklklklk" // Your Wi-Fi Pass

 

Ubidots client (TOKEN);

 

/****************************************

 * Auxiliar Functions

 ****************************************/

 

void callback (char* topic, byte* payload, unsigned int length) {

  Serial.print("Message arrived [");

  Serial.print(topic);

  Serial.print("] ");

  for (int i=0;i<length;i++) {

    Serial.print((char)payload[i]);

  }

  Serial.println();

}

 

/****************************************

 * Main Functions

 ****************************************/

 

void setup () {

  // put your setup code here, to run once:

  //client.ubidotsSetBroker("business.api.ubidots.com"); // Sets the broker properly for the business account

  client.setDebug(true); // Pass a true or false bool value to activate debug messages

  Serial.begin(115200);

  client.wifiConnection(WIFINAME, WIFIPASS);

  client.begin(callback);

  }

 

void loop() {

  // put your main code here, to run repeatedly:

  if (!client.connected ()){

      client.reconnect();

      }

 

  float value1 = analogRead(0);

  //float value2 = analogRead(2)

  client.add("temperature", value1);

  //client.add("status", value2);

  client.ubidotsPublish("my-new-device");

  client.loop();

}

 

 

This blog has been submitted by KRSSG, IIT-Kharagpur under the Robocraze Club Outreach Program.

Author: Moinak Ghosh