My project aims to create a program that will be able to instantly detect a person's mood and classify it using machine learning and change the ambience accordingly to make the person feel happier and calmer. It will aim to reduce the prevalence and mount up of common neurological depression, distress, and overwhelming anxiety. The program will adjust the ambience to improve the mood by using color and audio therapy to manipulate the colors and sounds to make the person feel better based on their personal preferance. 

Depression, stress, and anxiety are common neurological disorders that affect around 20-25% of the world's population today. Though these disorders start small, they can grow into an uncontrollable storm if not handled properly over time. Many factors can affect the development of these disorders. One of the main ones are environment and an example of this is Seasonal Affective Disorder (SAD). SAD is an neurological disorder where people start feeling upset or gloomy in the winter time. One of the reasons behind this in the lack of sunlight and Vitamin D in winter, and the lack of vivid colors and bright outdoor environments. The harshness of winter can make people feel sad and depressed, and give them more anxiety about small things. This shows the large influence of color and audio on people's mood. The ambience (color and audio) of someone's surroundings can help to lessen or even completely diminish the effects of these kinds of disorders.

Mood Sensors

Different Types of Sensors

• Here are some popular sensors which provide the necessary vitals to determine someone's mood:

• Sensor	Accuracy
  Electroencephalogram (EEG): Measures using electrical signals of the brain. Usually measures different wavelengths in the brain.​	~88.5%
  Electrocardiogram (ECG): Measures using electrical signals of the heart. Respiration Rate and Heart Rate Variability are usually measured using this. ​	~85%
  Galvanic Skin Response (GSR): Measures using changes in your skin. Skin Temperature Measurements can fall under this category.	~76%
  Gestural Measurement: Measures using changes in body posture and facial expressions. Usually measures using relations between emotions and gestures. ​	~72%
  Electromyogram ​& Electrooculogram (EMG & EOG): Measures using electrical signals of muscles. Usually measures using changes in face expressions and measures using the electrical signals of the eyes. Usually measures using movement in the eye and cornea. ​	~70%

   

  Gestural Measurement

• As seen in the table above, there are many different sensors to detect mood. EEGs and ECGs are some of the most accurate sensors today, but the sensor that I used for this project is gestural measurement, as highlighted in blue. Gestural measurement encompasses many different subsensors, such as facial recognition, voice recognition, and body posture. This mode of measurement is not the most accurate, however, as shown above.

• I chose this mode of measurement because it is the most practical mode in the real world compared to the ones with higher accuracy. In EEGs, ECGs, and GSR machines, the person has to have electrodes attached to their skin in various machines, and have very limited mobility. Imagine having to wear a EEG cap all the time, or have ECG electrodes all over your body. This makes EEG, ECG, GSR unrealistic and impractical in the real world. On the other hand, in gestural measurement, you only need camera footage or a microphone, which can be placed almost anywhere, and sometimes without even the person noticing that it is there, while still providing proficient results. Gestural measurement is the most seamless way to integrate mood detection into our daily lives.

 

 

Machine Learning

• Machine learning is where machines learn from data sets and generalize to unseen data. They use statistical algorithms to classify data into specific categories. There are many types of machine learning models that can be used to classify data. There are supervised models and unsupervised models. I am using a supervised model for my project, and out of the methods used in supervised learning, I am using the direct classification model to categorize the data. For this project, I am using Teachable Machine Image Model to detect mood. In this project the classes I have are 6 common emotions: Angry, Surprised, Happy, Calm, Sad, Scared.

 

1. Sense: The input is what data you are giving the machine learning model to classify and sort into a category. For example, lets say your input is a person smiling. The model will take this input, sense it using a sensor, which in this case is the camera, and produce an output depending on what it is trained on.

2. Analyze: The machine learning model takes the input given to it and analyzes it. It will use image recognition and pick up details in the input. It will then look at details to classify it. In this example, the model might pick up the corners of your mouth turned up, and your teeth showing. It might see your eyes slightly squinted. The model will also use statistical algorithms to determine the mood. It will take these slight hints and, refering back to its training, it will analyze it and classify it into an mood.

3. Classify: After the model analyzes the input it will determine which category the input fits into, and in this case it will determine the mood of the person. It will compare it with other datasets that it was trained with, and see which categpry is matches with the most. For this example, the model will compare the small details to other images in the Happy dataset, and classify it as Happy, and it will now accordingly change the ambience.

4. Improve: After the model classifies the input into a mood, the program will then change the ambience according to what mood the model classified. It will work to either uplift or relax the person, or work to keep the person feeling good. In this example, the mood was Happy, so the color will change to yellow and a bright song will play to keep the person in their joyful mood.

 

Color & Audio

The Influence of Color on Mood

• The science of color perception is an interplay of physics, biology, and physiology. It's what makes our experience of the world so rich and vibrant. ​

• Chromotherapy: An ancient healing practice that uses color and light to balance the mind and body. Ex. Egyptians utilizied sunlit rooms adorned with colored glass to aid in healing. It may also have roots in Eastern healing practices like balancing chakras, where each chakra (pools of energy inside the body) is associated with different colors.​

• Color is a powerful communication tool and can be used to signal action, influence mood, and influence physiological reactions in our bodies. 

• Understanding the physiological impact of colors allows you to harness their power. By choosing specific colors, you can create environments or select therapies that support mental health needs effectively. Colors play a pivotal role in shaping our experiences and emotions. Using color strategically can foster an atmosphere that supports specific tasks. Ex. Red and yellow can boost creativity, but too much intensity might be distracting. Subtler tones such as blue and green can enhance concentration and help maintain a calm environment, making them a popular choice for offices.​

• Physiological Response: The visual system perceives colors, different colors stimulate the retina and nervous system, for example, the color red is believed to increase heart rate and respiration, while blue has a calming and relaxing effect.

Light Perception

• Light enters the eye and hits the retina at the back of the eye. Inside the retina are rods and cones. While rods are used to help us see better in the dark, cones help us perceive colors that are reflected off objects and make our world so vibrant. Our ganglion cells take this color information and transform it into action potential to send down the optic nerve and to our brain, where it is converted into an image that we can understand.

• The different wavelengths of the colors evoke different responses when they hit our cones, and also when they hit other parts of our body, such as our skin. They stimulate different parts of our brain, therefore each color having a unique effect on our mood. 

• Fun Fact: Tetrachromacy is a rare genetic mutation where the eye has 4 types of cone cells instead of 3, allowing people with this mutation to see more colors than an average person. It is more common in woman than in men, and can affect the influence of colors on these people.   

 

The Influence of Audio on Mood

• Understanding of music's influence on brain is intricate, as it involves multiple brain areas and neural connections to process auditory information, emotions, and memories.​

• Music, a rich tapestry of sensory, cognitive, and emotional experiences, is not just about the melody or rhythm. Music evokes a spectrum of emotion, from joy and sorrow, to nuanced feelings, like nostalgia. This is a confluence of multiple brain areas, responsible for attention, motor functions, memory, intertwined with emotional and motivational pathways. The comprehension of harmony in music engages distinct neural domains including the auditory, prefrontal, and parietal cortices. Music's emotive power is rooted in the activation of the brains reward system including the nucleus accumbens and the ventromedial prefrontal cortex. ​

• Studies have shown that music has high rate of success for the treatment of depression, anxiety, and other neurological disorders using audio therapy. ​

• Slow beats can make us feel calmer, while faster beats can increase our focus because of the way the soundwaves hit our tympanic membrane and get sent to the auditory cortex.​

• "Music can be a drug". It increases dopamine in nucleus accumbens like cocaine. ​

• Music can control your fear, make you ready to fight and increase pleasure because of the activation of the amygdala. ​

• Hippocampus produces and retrieves memory, regulates emotional response and helps us navigate, and this region is the first region to be affected by Alzheimer's disease, leading to confusion and memory loss. Music may increase neurogenesis in hippocampus which improves memory. ​

• The putamen processes rhythm and regulate body movement and coordination. Music can increase dopamine in this area increasing the response to rhythm. By doing this, it can temporarily stop the symptoms Parkinson's disease. ​

Audio Perception

• Soundwaves will enter our ear and hit the tympanic membrane inside our ear and travel through a path of tiny bones and cartilage. It will then enter the cochlea, a snail-shaped fluid-filled organ, which is lined with tiny hairs. The cochlea will direct the soundwaves, and turn them into electrical signals that travel up auditory nerve to the brain.

• Each part of the cochlea is assigned to a different part of the auditory cortex, so when different frequencies hit the tympanic membrane, they reach different parts of the cochlea, and each part stimulates a different part of the auditory cortex, therefore stimulating different parts of the brain, proving that different types of music can each have a different effect on the brain.

 

 

Overview

• For this project, I am combining mood sensing with machine learning and color and audio therapy to be able to make a device which will encompass all of these different aspects to detect, classify, and improve a person's mood seamlessly and in real time. 

Final Product

• In the final product, here is the list of colors and songs I used for each mood.

• Angry: For the mood angry, I chose the color blue. Blue is known to have calming effects. It helps lower your heart rate and blood pressure, and helps to reduce stress. It is associated with tranquility and serenity, bringing peace over an environment which is filled with this color. Because of its calming properties, a grayish-blue color is usually used in office environments and hospital wards. The song I chose is "Someone Like You" by Adele. This song has a slow tempo and its consists of piano melodies. The slow tempo and beat aligns with our own heartbeat and helps to make us calm down, and make us feel more relaxed. The slow piano melodies are unconciously calming because of their sound and tempo, which makes us feel peaceful and calmer. For these reasons I chose the color blue and the song "Someone Like You".

• Surprised: For the mood surprised, I chose the color yellow. Yellow is often associated with positivity and happiness, though it is not very alerting or standing out compared to other warm colors like orange and red. It can automatically improve someone's mood just by being in an environment surrounded by this color. It can be used to let us soak in surprise in a positive way, while not being to alerting or contrasting. The song I chose is "Happy" by Pharrell Williams. This song is upbeat and has a fast tempo, while still being classy and not too loud or aggresive. It makes us feel more positive, while keeping our energy limits to an optimal level. For these reasons I chose the color yellow and the song "Happy".

• Happy: For the mood happy, I stuck with the color yellow. Yellow makes us feel happy, but it doesn't make our energy level increase drastically. It helps to keep us feeling happy when we are already in a positive mood. Yellow is a natural feel-good color. An example is sunshine, or sunflowers, which naturally make us feel happy. The song I chose is "Can't Stop the Feeling" by Justin Timberlake. This song has a natural optimistic vibe, and its fast and upbeat tempo and rhythm help make us feel happy, and keep our positive up when we listen to it. It has a catchy lyrics and it great to keep us feeling happy. For these reasons I chose the color yellow and the song "Can't Stop the Feeling".

• Calm: For the mood calm, I stuck with the color green. Green is known to be calming and peaceful calming. Its association with the natural world, such as leaves and grass, makes it a calming color, as nature is often represented as a tranquil environment. This makes green a good color for a peaceful environment, bring tranquility and serenity to the environment. Green is also known as a positive color in Western culture. Ex. Green means go, 100% is usually green. It helps us remain in a calm mood if we are already feeling relaxed and stress free. The song I chose is "Interstellar Main Theme" by Hans Zimmer. This song has a very peaceful, yet meaningful vibe, making it perfect for calm environments. It encompasses many different instruments, and composes a very calming musical piece. The different instruments makes it tranquil, and keeps you feeling calm when listening to it. Its overall composition makes it ideal for peaceful environments. For these reasons I chose the color green and the song "Interstellar Main Theme".

• Sad: For the mood sad, I chose the color orange. Orange can be used as a color for caution, or as a color to gradually uplift our mood, but not to the extent of the color yellow, where we get very optimistic. If used in the correct shades and context, it can help create an overall subtly positive vibe, and can be used to help us feel better after feeling sad or upset. It can help us ease our way out of sadness and into joy in a gradual way, without being too overpowering or sudden. I chose the song "Shake It Off" by Taylor Swift. This song has a positive vibe, but does not automatically jump into an upbeat theme. It makes us feel better, and its fast tempo and rhythm make us feel more optimistic, and helps bring us out of sadness. For these reasons I chose the color orange and the song "Shake It Off".

• Scared: For the mood scared/anxious, I chose the color green. Green in calming, and can help reduce stress in environments where it is used. It is often associated with nature, and delivers a similar effect of peace and serenity. This can also be used as a grounding color, helping us asses the size and importance of a problem realistically. It helps make us feel less anxious and gives us a reassuring feeling. It can also help make our overall mood calmer and may help us hone in our focus and work efficiently with positive stress, instead of distress. The song I chose is "Titanium" by David Guetta and Bebe Rexha. This song has a very motivating lyrics and helps make us feel more confident. It is quiet and slower in some parts, and cresendos up to a faster and louder part. This makes us calmer, while still making us more concentrated and confident. It variety of tempos and instruments help make us reduce stress, while making us feel more positive. For these reasons I chose the color green and the song "Titanium". 

• Overall, for each mood, there is a specific color and song allotted to improve the mood, each with a scientific explanation behind why I chose them. These use color and audio therapy to improve the mood. 

• Some limitations to this project were the lack of access to advanced mood sensors and to advanced webpage design. ​

Overview

• This project overall is a non-verbal mood indicator and mood improver by combining color and audio therapy with the power of machine learning. ​

• It helps people express their mood in real-time and in a natural, automatic and seamless way, instead of awkwardly having to explain to a person how you are feeling. It helps to get context, and it can also be used on others entering a person's environment to help match their mood to some extent, depending on how they are feeling. ​

• It can also help in reaching an optimal mood. It will detect the person's mood and adjust their ambience to help uplift their mood or help them calm down, depending on how they are feeling. If they are already at an optimal mood, it will aim to keep them in that good mood by changing the ambience to keep their mood up. ​

• This project is innovative because it uses machine learning to detect mood. It can be taken to the next step by adding better input devices in larger numbers and using AI to help determine mood in a more accurate way for the specific use. It could also be improved by adding output devices, such as RGB lightbulbs and speakers around the area to help improve ambience. A self-control aspect can also be added to this device to allow people to customize their ambience to be more effective. ​

Future-Case Scenarios

1. Rhythms of the Road​: Envision a future where automobiles are equipped to sense the correlation between the music being played, driver’s mood, and assess the risk of accident based on the speed of the vehicle and weather conditions.​ Automobiles are being increasingly equipped with computing systems that are capable of complex calculations and act in a semi-autonomous and fully-autonomous way to enhance the safety and experience of passengers.​

2. Retail Application: Imagine a retail store, where the music playing and the colors of the walls could have a large influence on shoppers. The music and colors could unconsciously uplift shoppers' mood, convincing them to spend more time in the store, browsing through items. ​The more time shoppers spend browsing through the store, the more chances there are of them finding something they like and buying it. Their uplifted mood also plays a role in their decision to buy something. This technique increases retail companies' sales largely in the long run. ​

3. Classroom of the Future: Imagine a future where classroom is more immersive and can dynamically transform into the setting of the subject matter. Depending on what topic students are studying, the whole classroom, such as the walls and audio, would change to accommodate their learning. For example, if students were reading a textbook on surgery, the whole classroom could change into an operation theatre.​ This approach to learning could make it much more engaging and effective compared to the traditional method of reading textbooks or listening to lectures. It makes it more exciting for both visual and auditory learners, enhancing their understanding of the topic.​

 

https://pmc.ncbi.nlm.nih.gov/articles/PMC7037130/ 

https://teachablemachine.withgoogle.com/ 

https://renewedmentalhealthgroup.com/what-you-need-to-know-about-different-types-of-moods/

https://github.com/

https://eqradio.csail.mit.edu/

https://journals.sagepub.com/doi/full/10.1177/2158244014525423#bibr1-2158244014525423

I thank my science fair coordinator for helping me set up my project, and guiding me throughout the year.

I thank my uncle for providing support and guidance when I was making my project.

I thank my parents and my sister for providing feedback and helping me improve my project.