Progress Report: 4/21/16

The Multi Layer Perceptron is working much better now. We found that randomizing the order in which the EMG points go into the text file that is read actually have a large impact on the accuracy of the code. Our randomizing process was in Excel, where we numbered all of the data in another column 1 through 5. The data was then reorganized by that new number so that the data was shuffled. This process was repeated for several more columns. At the end of this, the data was put back into MLP, and this resulted in a much lower guess error. It went from 80% to 8%. This is a large accomplishment and means that we can successfully classify data and we can also successfully create more data to increase the accuracy of the MLP.

Challenges we have come across were the high error rate of the Multi Layer Perceptron which we had initially. Another issue is that the data that we are using is fake data based off of the limited amount of real data that we have, so the high success rate may be a result of the fact that the data is easier to guess when it is artificially created rather than when it is constructed. When we gather more data, we shall see.

At this point in time, we now have the Multi Layer Perceptron working, so our next steps will be to gather more data and preferably to not rely on point generation. We will try and use multiple EMG Spikershields at the same time in order to gather more accurate, real time, data. Another thing that we would like to do is to streamline the entire process that we have, as at the moment every step is separate and time consuming, and it is all split up between different programs and software. If we could get all of the steps to happen at once, that would be a large improvement over what we currently have.

Progress Report: 4/9/16

We have made progress in applying the data that we currently have to the multi-layer perceptron. We took time to understand the code for single and multi-layer perceptrons. After this, we put our code into the perceptron to see how well it could guess what finger was moved based off of gathered data. To do this, we had to create a code that took the data that we had and organized it. From there, the code would find the maximum and minimum in the data, and convert it so that it was between -1 and 1 to be used for the multi-layer perceptron. We put the data into the trainer and tested it. We also tried to interpolate data, making up similar data points to give our perceptron greater reliability.

Some problems we encountered were that the multi-layer perceptron is not very accurate with the data that we have. There are several factors that could have hurt its accuracy, but the likely one is that the data that we have is simply not enough to adequately train the code. Another problem we faced was the interpolation of the data, which seemed to be all over the place, perhaps indicating that the data we have gathered is not precise enough. We have also had trouble getting into Matlab, since the trial expired, making it hard to gather more data.

From here, we intend to figure out why we cannot access Matlab and continue to gather more data to be used. We also want to figure out why the interpolation is so random, and if that is due to a problem with our data.

Progress Report: 4/5/16

This past week was presentation and working on our abstract so we did not advance much with our project. We have been trying to get MatLab to work again as it is not working now. This week's goal is to record another muscle to help further the differentiation.

Our abstract: 

The purpose of our project, Hand-to-Hand Communication, is to use EMG signals within our arms to mimic movements on another individual/robotic arm. Today, we will demonstrate how we take raw data and process it. We find the highest values, the peaks for each muscle movement, using a code, and then graph those values. This process is repeated with other muscles, in the attempt of finding a pattern.

Progress Report: 3/16/16

https://drive.google.com/a/erhsnyc.net/file/d/0Bx2jM7Lyt10odEtXekxPT3E3c1k/view?usp=sharing

These data consist of x as one muscle and y as the other. The muscles are at the bottom of each picture.

Diagrams of Recorded Muscles

So far, we have recorded EMG signals from three fingers. Below are the diagrams for the three muscles we have recorded, highlighted in green. They are extrinsic muscles, responsible for the extension of fingers but located in the forearm.

Extensor Digiti Minimi: Responsible for pinky extension.

Extensor Digitorum: Responsible for the extension of the four fingers, not including the thumb.

Extensor Pollicis Brevis: Responsible for the extension of the thumb.

Progress Report: 3/10/16

For the past few weeks, We have been collecting data for the different muscles that we need to analyze. So far, we have recorded the data for all five fingers for Extensor Digiti Minimi and Extensor Pollicis Brevis.

These past few weeks we have been also working on finding a way to separate the points

This is the graph we gathered. The x coordinate was Extensor Digiti Minimi and the y coordinate was Extensor Pollicis Brevis.

From the graph, we can see that there is a clear difference between the thumb and the other fingers. This is what we predicted because the thumb's movement is caused by an entirely different muscle while the other fingers are all intertwined and are moved by common muscles. Now that we have this, we are going to work on differentiating the other fingers so we have to now start dealing with 3D and 4D.

https://drive.google.com/a/erhsnyc.net/file/d/0Bx2jM7Lyt10oelZETVpDTFF2Rm8/view?usp=sharing

EMG Muscle Control Devices

Here is the video about a muscle control device I mentioned in class. It is very similar to your project. You may be able to find more valuable info through this thread. Good luck!

Progress Report: 2/7/16

Since the last progress update we have made a lot of progress. Firstly, we researched the extrinsic muscles of the hand that are responsible for finger movement. We created a table that covered all of the necessary information, which would allow for more precise placement of electrodes and thus better data gathering. We have gotten Matlab to work, using code that we found online. The code forms a graph from the data that comes in through Arduino, and we saved the numerical values. We have been recording data from the extensor digiti minimi because it is the only muscle that we could find that was both superficial and responsible for only one muscle movement. We recorded each finger's extension from that muscle with the hopes of finding a pattern or seeing how much interference is involved. After recording the data in each finger, we found the maximum points from each finger extension, and then found the average value of that maximum EMG reading, which we did in Excel.

Some problems we encountered were what to save the Matlab workspace as, since we saved it as a .mat rather than a .m. Another problem we have encountered is that a lot of the muscles that are clearly defined as affecting only one finger are deep muscles, meaning it will be harder to record data. Another problem is that certain fingers don't have a unique muscle that we can measure. We also are limited by the fact that we have only one recorder for EMG signals, since we only can record one channel at a time.

Our goals are to figure out which finger we are going to record next, and also to do some coding in Matlab to try and cut down on the amount of time that it takes to get the maximums from Matlab automatically.

From Arduino to MATLAB

EMG signals acquired by the SpikerShield and Arduino Uno can be send to the Macbook Air through serial port (USB). The current problem is how can we read and even display the data from MATLAB. After trying a few possible solutions from the internet. There are two working solutions you can base on for your next-step development:

1. Serial Monitor (Debugger): It can be downloaded from http://www.mathworks.com/matlabcentral/fileexchange/45839-serial-monitor--debugger-. It is an app that you can install, and then run it by clicking the APPS tab -> MY APPS -> Serial_Monitor. It will show the serial data in a pop-up window. You can save the data to the "SerialData" variable for further processing.
2. Real Time Serial Data Logger: It's a piece of code that you can run in the MATLAB. It can be copied from https://billwaa.wordpress.com/2013/07/10/matlab-real-time-serial-data-logger/. You can just copy and paste it into the MATLAB, and a pop-up Serial Data Log window will appear to show the plotted serial data.

The followings are the screen shots and videos of these two solutions in action.

Serial Monitor (Debugger)

 

Serial Data Logger

 

 

Finger Gestures vs. EMG Electrode Placement

Congrates to be able to gather raw EMG data from the SpikerShield! While you are picking up MATLAB, we also need to figure out the EMG electrode placement as soon as possible such that we can start acquiring meaningful data. There are three documents in our the Project Resource page, which potentially provide us the info we need.

1. Finger Motion Decoding Using EMG Signals Corresponding Various Arm Postures (2010), Kyung‐Jin You, Ki‐Won Rhee and Hyun‐Chool Shi
2.  Relating Forearm Muscle Electrical Activity To Finger Forces (2014), Jennifer Keating. [page 22-23, 62-84]
3. Forearm Muscle Labels.

The simple questions we want to start with are: "In order to detect the movement (bending and stretching the finger) of individual finger,

1. what are the most important muscle(s) related to the movement (bending and stretching) of individual fingers?
2. How many electrodes (or electrode pairs) do we need? And,
3. where should we place those electrodes (or electrode pairs)?"

Can your team look into them, extract useful info, and summarize the results into two tables (bending and stretching)? The tables should at least consist of the following columns: Finger, Muscles (names & symbols), Category (deep or superficial), Type (extension or flexion), Location (anterior or posterior), Electrode Placement (labels matching the diagrams), and Comments.

 

The exact electrode placement (position and orientation) should be labeled and indicated in the following tow diagrams (anterior view & posterior view) and matching the electrode placement in the tables.

Forearm muscles of anterior compartment: superficial, middle, and deep

Forearm muscles of posterior compartment: superficial, middle, and deep

You can contact me through email/blog whenever you have any question. Don't forget to post your results onto your blog and we will discuss our next step (data acquisition) based on the results after the break.

Progress Report: 11/30/15

We were having a lot of trouble getting MATLAB to work on the school computer, but we managed to download it onto one of our computers. In addition, we have figured out how to gather data on Arduino. We then plotted it using the spreadsheet application on Google Drive. The picture below is just a proof of concept to show that we can gather data. The values are a sign of strength of muscle movement, and each data point is collected every 100ms. The graph is for a very long time, being about 16 and a half seconds.

Some problems we faced this week were getting Matlab to connect to Arduino on the computer to which we downloaded Matlab. For some reason it didn't work. Another problem we encountered was understanding Matlab, as looks like we will have to gain an understanding of coding and algorithms.

The plan for us is to do isolated tests of each finger for a much shorter period of time. We will repeat a specific motion, targeting a specific muscle, and through this, hopefully we will be able to identify patterns through this approach. We also hope to gain a better understanding of how to use Matlab through tutorials. We will also try to learn some coding in Arduino, although it looks like this will not be as necessary as previously thought. We have made some progress this week, we feel, and are excited to continue.

STEM Patent Research: 11/30/15

An excellent list of patents, and nice summary and analysis. Based on your description in "PLACES WE CAN INNOVATE", you might want to look further into a field called Functional electrical stimulation (FES) or NeuroMuscular Electrical Stimulation (NMES).

STEM Patent Research: 11/30/15

PROCESSING EMG:

https://www.google.com/patents/US8280503?dq=emg+signal+communications+hand&hl=en&sa=X&ved=0ahUKEwiz66H62qnJAhVGSSYKHR9XAFcQ6AEIGzAA

Assessment of EMG signal acquired from parts of the body where there is a lot of activation of muscles involved in motor skills (ability to do complex muscle and nerve act to produce movement). It can compare the pattern observed to known patterns obtained during a controlled activity.

https://www.google.com/patents/US20080200827?dq=EMG+signal&hl=en&sa=X&ved=0ahUKEwiD8Mju5KnJAhXBWz4KHbYcBhEQ6AEIRTAG

The invention is a circuit that can take EMG signals and transfer them via USB to a computer.

https://www.google.com/patents/US6411843?dq=EMG+signal&hl=en&sa=X&ved=0ahUKEwiD8Mju5KnJAhXBWz4KHbYcBhEQ6AEIMDAD

A way and apparatus that can be used to produce a model EMG signal from a measured one by a series of filters. The EMG signal that results is separate from the EMG and EKG signal that is measured.

https://www.google.com/patents/US8280503?dq=emg+signal+hand&hl=en&sa=X&sqi=2&pjf=1&ved=0ahUKEwjm76-d0rnJAhXKOz4KHYVhAFgQ6AEIHDAA

EMG signals are detected from several different locations on the hand. These locations of electrodes are where precise movements occur. The EMG signals are registered and processed to be used for biometric assessment.

https://www.google.com/patents/WO2012142241A2?cl=en&dq=emg+communications&hl=en&sa=X&sqi=2&pjf=1&ved=0ahUKEwj806Kk37nJAhUHpB4KHTuRAxsQ6AEIIjAB

Acquiring uterine EMG signals. Signal processing device transmits the signal to a relaying device which sends it out to a call center for a doctor to look at. Can potentially help us figure out how to transmit the signal to another arm etc. At least one pair of electrodes used.

ANALYZING EMG:

https://www.google.com/patents/US8447704?dq=emg+signal+communications&hl=en&sa=X&ved=0ahUKEwjFkNC12qnJAhWJSCYKHcz8D1c4ChDoAQgjMAE

A machine learning model that has the user do specific gestures so that it learns the signals from that gesture. The machine learning model can then identify specific gestures from specific fingers of the user using the information that it has learned.

https://www.google.com/patents/WO2009140149A2?cl=en&dq=transmitting+emg&hl=en&sa=X&ved=0ahUKEwjdw6_q37nJAhXLRiYKHfCrAsIQ6AEIRTAG

A sensing device that allows patients to control an object by using motor unit action potential. It uses an emg sensor which they place on a specific area of the patient. It also configures a signal that represents the motor unit action potential. Then it uses a personal area network transmitting device  that corresponds to a specific signal. Personal area network is used to convert it to an electrical signal which then goes through a processor where the electrical signal is received to generate at least one control signal.

GADGETS TO HELP HUMANS:

https://www.google.com/patents/EP2865356A1?cl=en&dq=emg+signal+communications+possessed+hand&hl=en&sa=X&ved=0ahUKEwiJiO6w26nJAhUDHB4KHRV3D-0Q6AEIGzAA

Hybrid prosthetic arm controlled by surface EMG (determines electrical activity of the muscle) (sEMG places electrodes on the skin overlaying the muscle and not INTO the skin) and mechanical control from elbow and shoulder of amputee. Device contains mechanical fingers driven by mechanical motors controlled by microcontrollers (a small computer (SoC) on a single integrated circuit containing a processor core, memory, and programmable input/output peripherals.) The instruction sent to the arm is from sEMG signals. It can convert sEMG data into instructions, and movement comes from motion of the shoulder, rotation of the elbow, and sEMG signal.

https://www.google.com/patents/US8828093?dq=emg+signal+identification&hl=en&sa=X&ved=0ahUKEwjglJiBybnJAhUB2T4KHZ3bAP4Q6AEITzAG

A lower prosthetic limb that uses EMG signals to determine the user’s specific gait phase. The device recognizes which EMG signals correspond with which type of locomotion to better replicate the user’s motion.

https://www.google.com/patents/CA2818146A1?cl=en&dq=emg+signal+hand&hl=en&sa=X&sqi=2&pjf=1&ved=0ahUKEwjm76-d0rnJAhXKOz4KHYVhAFgQ6AEIODAE

An athletic glove that records EMG signals from electrodes on the inside of the glove. The data is gathered and processed, and then output on an external module of the glove. This data provides feedback on grip to the user.

https://www.google.com/patents/US20020019652?dq=tens&hl=en&sa=X&ved=0ahUKEwjSkduh4rnJAhVFwj4KHf9eACUQ6AEIPzAF

TENS bandage that uses electrical stimulation to ease and block pain within wounds. Also helps with healing of a wound.

https://www.google.com/patents/US20020077689?dq=tens+control+muscle&hl=en&sa=X&ved=0ahUKEwiby9D847nJAhWBQD4KHa9YB7MQ6AEIIzAB

A suit that has electrodes placed on it. Wires connect the electrodes together with a form of stimulation device. The wearer or user can apply electrical stimulation to certain muscle groups or parts of the body, inhibiting a response.

STIMULATION:

https://www.google.com/patents/US8571662?dq=nerve+stimulation&hl=en&sa=X&ved=0ahUKEwi_wrTP47nJAhUK5iYKHeFzCMIQ6AEIGzAA

Stimulating nerves using electrodes with an electrically insulating back layer. Increases electrical current through surrounding tissues. Increases impedance of electrical path through blood in lumen of blood vessel.

https://www.google.com/patents/US7725193?dq=emg+tens+unit&hl=en&sa=X&ved=0ahUKEwix366G47nJAhVMlh4KHUh4AUwQ6AEIGzAA

A surface probe with a conductive tip that can apply a local high voltage. The electricity applied causes a stimulation of the targeted muscle fibers, eliciting a forceful movement from those muscles. The force and number of twitches can be altered.

https://www.google.com/patents/US20100318009?dq=tens+unit+hand&hl=en&sa=X&ved=0ahUKEwiO44ub4rnJAhXEOT4KHQ5aCpsQ6AEIKjAC

A device that can send electrical stimulation to the extremities specifically. It braces the hands in place on a flat surface and then provides the shock with a TENS unit.

https://www.google.com/patents/US8565888?dq=tens+control+muscle&hl=en&sa=X&ved=0ahUKEwiGj93p5LnJAhXMdD4KHUFXAwkQ6AEIJDAB

An automated system that is able to deliver electrical stimulation to the user. It can then also detect the muscle response from the electrical stimulation. Being able to detect muscle response and deliver electrical shocks, the system is able to automatically diagnose one characteristic of a muscle from the response and adjust the electrical stimulation accordingly.

https://www.google.com/patents/US8620434?dq=tens+precise+stimulation&hl=en&sa=X&ved=0ahUKEwjV6eyq0rrJAhUJRiYKHc9gAoUQ6AEIHTAA

A TENS device that is able to apply electrical stimulation to muscles and also detected changes in the skin. There are two modes, stimulation mode and re-calibration mode. Via a plurality of electrodes, the device can apply an adjusted electrical current to the user based on skin impedance.

https://www.google.com/patents/CA2132157C?cl=en&dq=MUAP&hl=en&sa=X&ved=0ahUKEwiq5c-B1LrJAhXJYyYKHSYgAMAQ6AEIJDAB

A device that contains a pulse generator to mimic MUAPs that are naturally generated. The invention can synthesize the basic wave form, not depending on muscle contraction to achieve any form of stimulation.

https://www.google.com/patents/CN104436429A?cl=en&dq=tens+unit+communication&hl=en&sa=X&ved=0ahUKEwjx7c793rnJAhWD6x4KHScdBNQQ6AEIGzAA

A TENS unit can connect to a smartphone via bluetooth. It can receive data as far which signals to transmit, and it can also send data to the smartphone in response to biofeedback from the user’s body. The controller can change pulse width, frequency, and/or intensity.

PLACES WE CAN INNOVATE:

The devices of EMG and TENS remain generally separate in the patents that we found. Both technologies seem to be known, it is just a matter of finding a method of combining these technologies. One place we can innovate is in a device that integrates both technologies together.

Another area that we can innovate is in TENS. Some of the patents that we found were able to adjust the signal, like in frequency or intensity. We could find a way to develop a TENS unit that can accept a changing signal and shock the user with that type of signal.

Virtual Reality Application

Hand-to-hand communication project covers two major technologies: decoding the EMG signals and encoding the TENS signals. Each technology has it's wide applications. Here shows a recent application in virtual reality (VR). Please see the YouTube video or read the article.

Progress Report: 11/22/15

A problem we were having was with getting the Arduino application on the computer to show the serial monitor. After a bit of trial and error, we got the monitor to load, so now we can have numerical data to use in deciphering different EMG signals. Because we have data in the form of numbers for the strength of the EMG signal, and the intervals in which the data comes, we can visualize the data, putting it into excel and creating graphs. We also found on the Backyard Brains website a pack of 100 electrodes, since we are running out of the electrodes that came with the Muscle SpikerShield Bundle.

We will need to learn how to filter out extra noise gathered from the SpikerShield. To do this we have to learn stuff about Arduino coding. We still are having trouble in gathering data, because the lights that light up on the SpikerShield seem to be inconsistent, and the electrodes also eventually lose functionality.

Our plan for the next week is to look at examples of Arduino code and gain an understanding of how we can approach filtering out noise from the signals that we gather. We also are going to work on getting real data from the SpikerShield. We are also going to look at the other patents on the subject for the patent project.

Progress Report: 11/15/15

We analyzed out SpikerShield box. We ran a test on the SpikerShield box and tried to figure how different moments affected the light popping up on the box (The light indicated muscle activity). We looked at the Arduino code and adjusted it a little bit.

Here you can see the lights are lit up due to his movement.

We had trouble accessing the monitor that showed our data (muscle movement) on Arduino so we had to troubleshoot and also MatLab is still not working on the computer. We might have to find a different software or spend more time on just MatLab. However, we decided to wait to work with MatLab since we need to focus on acquiring a signal.

The plan is to figure out how to display the monitor that acquires our data through Arduino. We also want to find more electrodes that we can use as our electrodes are not reusable and don't last very long.

Progress Report: 11/8/15

As far as accomplishments, we finally got the Backyard Brains Muscle SpikerShield bundle. We have been researching and examining the equipment, looking at the materials online regarding the SpikerShield. There are different experiments that they have to try out and we have been reading through them. We have also worked on installing Matlab, and have found a download that will help us to install the software and get started with Matlab. We also downloaded the software for the SpikeRecorder software from Backyard Brains, and tried to get it to function with the SpikerShield.

Some problems we faced were in installing Matlab. Due to the way the software uses the internet and the various firewalls and blocked websites that exist in the school, it is very hard to install the software. We also encountered problems with the Backyard Brains app. We plugged the SpikerShield into the computer and tried to find a way to get the app to switch from the computer microphone to the SpikerShield itself. We have been unsuccessful in it so far.

We plan to find a work around for Matlab this week, and also perform the first actual test with the SpikerShield at home on Wednesday. Because we want to minimize noise and perform the test in a controlled environment.

Progress Report: 11/1/15

We were able to accomplish one of the most important parts of our project this week. We were able to do all of the final product research on finding the right equipment (Backyard brains) for what we are trying to accomplish with our project. We placed the order this week and we should receive the product sometime next week. We also looked into whether or not we could use the Uno board - Leonardo instead of the Uno and it turns out that we could so we are currently working on that.

We tried to figure out Matlab by downloading the actual program onto our computer but we were having trouble setting up the program on the computer and Matlab seems to think it is a proxy error so we have to look into it and/or contact MatLab

Our plan for next week includes trouble shooting MatLab, trying out the equipment assuming it is delivered and also looking more into the program for the Arduino board.

Progress Report: 10/25/15

We have decided that the most feasible product is something simple that has integrated software that if need be we can upgrade in the future. Looking at the brainwave groups, which likewise used simpler tools as the starting point to tackle a more advanced problem, we decided this would be the best approach. We have chosen the Muscle SpikerShield from Backyard Brains, since it is relatively cheap and can be upgraded to get up to 5 channels. We also set up an account for Matlab, a data software. 

We still need to place the order with approval from Mr Lin, and this aspect only addresses parts 1 and 2 of the project. We haven't found a TENS unit, but at the moment we are looking to just start. 

From here we will research the equipment that we have chosen, reading manuals and things before it comes. We might also look into how the Arduino works, since that is a part of our project. We should have it by next week at the latest, and once we get it we will begin to run tests on how it works. 

https://backyardbrains.com/products/muscleSpikerShield

https://backyardbrains.com/products/spikerecorder

https://www.arduino.cc/en/Main/Software 

https://www.mathworks.com/programs/trials/trial_options.htmlprodcode=ML&eloqua_redirect=true&product_info=ML&verified 

https://backyardbrains.com/experiments/muscleSpikerShield

RE: Team 7 Project Gantt Chart 9/24/15-11/8/15

Feedback:

1. Go ahead to start your project accordingly!
2. It takes too long to research equipment.
3. You can group ordering and shipping together.
4. The granularity of "Research Muscles and Nerves" needs to be reduced by dividing them into sub-tasks.
5. There are many overlapping tasks. You need to assign resource in order to see whether it is feasible.