“Lifelogging” and neurophysiological computing: Will we forget how to forget?
One of the most famous examples of reminiscence includes a
madeleine dipped in tea, which lead to almost 3,000 pages of recollection by
the narrator in the beginning of Marcel Proust's novel In Search of Lost Time,
and we have all experienced these sensory triggers to a particular memory. Remembering
the past helps us to re-examine our lives, make choices, and share personal
accomplishments. We often use external devices to help us remember
events big and small, and with advances in technology, we often record and make
plans using a variety of digital devices such as iPhones, Microsoft’s Outlook,
and even smart watches. We have the capability to store a lifetime of data with
these advanced technologies, and with the advent of Facebook, Twitter, “selfies”,
and blogs it has become routine for many people to document their lives on a
daily basis in a digital form, a practice that has been referred to as
“lifelogging.” The outcome of documenting activities digitally are human
digital memories (HDM), which have been defined as “a combination
of many types of media, audio, video, images, and many texts of textual content [1].
The concept of recording and then later having the ability
to review certain documents was first proposed by Dr. Vannevar Bush in 1945
when he described the “Memex”
(a combination of “memory” and index”) in an issue of Atlantic Monthly [2]. As described in the article,
a Memex was “a device in which an individual stores all his books, records, and
communications, and which is mechanized so that it may be consulted with
exceeding speed and flexibility. It is an enlarged intimate supplement to his
memory.”The device would look like
a desk where documents were either recorded via microfilm or photography.
Since that time, many
similar devices have been developed, but a revolutionary advance was seen with
Microsoft’s SenseCam a wearable camera with a wide-angle lens and multiple
sensors, including an infrared sensor to detect the presence of other people.
The camera takes a photo every 30 seconds, resulting in up to 2,500 photos a
day and is capable of storing 30,000 images in total. Photos can then uploaded
to a computer and viewed later using a Microsoft application [3]. SenseCam was developed as a memory aid and there has been over 50 research institutions
that have used the device in a variety of studies involving memory and behavior [4]. Notably the SenseCam has shown promising results in studies where it
was used as memory aid for a child with anterograde amnesia [5] and with adult patients that were
suffering from amnesia [3]. Aside from the medical
purpose that a camera such as SenseCam
could potentially serve, “lifelogging” has become more socially acceptable as
we live in a digital age where Facebook posts and Twitter feeds are consumed
constantly, and “selfies”
are a regular occurrence at most events.
However, our memories and our experiences are made up of more
than static images. Our memories are composed of sensory information, such as
temperature and smells, and especially emotions and physiological signals. The
next step of HDM would involve going beyond just digital images, and instead
would include physiological information that has been captured with sensors.
This type of information could allow for a more vivid recall and potentially
could remind us how we felt at any point in time [4]. Wearable systems that
incorporate sensors which collect data for future review have been the focus of
many researchers. One example of this type of system is the “Physiological Sensor Suite (PSS),” which
collects electrocardiogram (ECG), electromyogram (EMG), electrooculogram (EOG)
and through-hair electroencephalogram (EEG) that is then sent wirelessly to a
data sensor [6]. Although the sensors of this
suite did not require a gel to be applied on the skin first, the most
effortless way to use sensors to record our daily activities would be smart
fabrics, such as “smart shirts” that combine textiles and wireless sensors
to monitor heart rate, angle of inclination, body temperature, and location [7]. Shirts of this type were originally
designed for a hospital setting as a noninvasive method to monitor patients,
but in the age of personal computing, technologies like this could potentially
be used outside of a hospital.
Of course, these
types of sensors that record physiological data such as heart rate or body
temperature would be essentially meaningless alone; it would be difficult to
reconstruct any memory, even a simple memory, based on physiological data since
it is so ambiguous. However, if physiological data were supplemented with more
data, such as a photo of the user, the location where the data was recorded,
and the temperature, these clues together may help to trigger certain memories
or parts of memories. One example of a system that works to combine data from a
variety of sources is the AffectAura, an emotional prosthetic
where data is collected from devices such as a microphone, Microsoft’s
Kinect, and a webcam to predict emotional states such as engagement,
valence, and arousal. Six participants were recorded over 4 days, and based on
the data collected, users were able to reconstruct stories about their days [8]. There are still multiple
challenges associated with creating accurate systems that could reveal
emotional aspects of a memory, especially when systems and devices are created
that go beyond only capturing photos. However, sensors will only become smaller
and smaller in the coming years, and as a society we have a great interest in
recording events for cultural reasons, so it is not unreasonable that one
point, most people will participate in the act of “lifelogging” and the
creation of HDM.
It is important to note though that physiological data or
even photos can only act as triggers to a memory, as there is no method to
actually capture an exact “memory” or a “thought.” Additionally, not only would
an HDM need to incorporate data from a variety of sensors and then correctly
corroborate and translate this data into meaningful information (a significant
challenge), but an entire lifetime of
memories would need be recorded for an accurate reflection and then a database
that is searchable would also be required.
If however we could accurately record and then disseminate
data that could compose a memory, does this documentation act as crutch?
Reminiscing and sharing personal stories with families and friends is a basic
human experience that acts as way to connect with others. If instead of
memorializing a lost relative through stories or laughing with friends over a
childhood experience, we could just push “play” on a device, how would that
change us? We already live in a society where any question can be answered with
a quick Google search on a phone requiring no discussion between two people, but
how would our interactions with others change if we could just “Google” how a
past experience played out or made us feel? (MIT professor Dr. Sherry Turkle has been studying how
technology impacts people for over 15 years and has written numerous articles
and books on the topics, and given this recent interesting TED talk on
the subject). Devices that use information from HDM are meant to help us with
the reminiscence process, but what if these devices are actually making us lose
that ability, or at the very least, fundamentally altering the memory process?
Additionally,
just as remembering is central to our existence, so is forgetting. Just because
at some point in the future we may be able to document a person’s entire
lifetime with a wealth of data, including physiological data, should we? Just
this year, researchers from the University of Basel discovered the musashi
protein, a protein that appears to inhibit molecules that stabilize synaptic
connections. These connections are important for the development of memories,
and based on this discovery, it appears that forgetting
is an active biological process. The biological processes behind
remembering and forgetting appear to work together, and forgetting is not just
a passive process [9]; we most likely forget for a
reason, even if more research is necessary to discover why.
References
1. Kelly L. The Information Retrieval
Challenge of Human Digital Memories. Proceedings of the 1st BCS IRSG Conference
on Future Directions in Information Access [Internet]. Swinton, UK, UK: British
Computer Society; 2007 [cited 2014 Aug 19]. p. 17–17. Available from:
http://dl.acm.org/citation.cfm?id=2227895.2227913
2.Bush V. As We May
Think. The Atlantic [Internet]. 1945
Jul [cited 2014 Aug 19]; Available from:
http://www.theatlantic.com/magazine/archive/1945/07/as-we-may-think/303881/
3. Hodges S, Williams L, Berry E, Izadi S, Srinivasan J, Butler A, Smyth G, Kapur N, Wood E. SenseCam: A
Retrospective Memory Aid. UbiComp 2006 4206:
177-193.
4. Fairclough, S.H., and Gilleade, K. (2014). Capturing Human Digital Memories for Assisting Memory Recall In Advances in Physiological Computing, S.H. Fairclough, and K. Gilleade, eds.(Springer London), pp. 211-234.
5. Pauly-Takacs K,
Moulin CJA, Estlin EJ. SenseCam as a rehabilitation tool in a child with
anterograde amnesia. Mem Hove Engl.
2011, 19(7): 705–12.
6. Matthews R,
McDonald NJ, Hervieux P, Turner PJ, Steindorf MA. A wearable physiological
sensor suite for unobtrusive monitoring of physiological and cognitive state. Conf Proc IEEE Eng Med Biol Soc 2007, 2007:
5276–81.
7. López G, Custodio
V, Moreno JI. LOBIN: E-textile and wireless-sensor-network-based platform for
healthcare monitoring in future hospital environments. IEEE Trans Inf Technol Biomed Publ 2010, 14(6): 1446–58.
8. Mcduff D, Karlson
A, Kapoor A, Roseway A, Czerwinski M. AffectAura: An Intelligent System forEmotional Memory.
9. Hadziselimovic N,
Vukojevic V, Peter F, Milnik A, Fastenrath M, Fenyves BG, et al. Forgetting Is
Regulated via Musashi-Mediated Translational Control of the Arp2/3 Complex. Cell, 2014, 156(6): 1153–66.
Want to cite this post?
Strong, K. (2014). “Lifelogging” and neurophysiological computing: Will we forget how to forget? The Neuroethics Blog. Retrieved on , from http://www.theneuroethicsblog.com/2014/08/lifelogging-and-neurophysiological.html
madeleine dipped in tea, which lead to almost 3,000 pages of recollection by
the narrator in the beginning of Marcel Proust's novel In Search of Lost Time,
and we have all experienced these sensory triggers to a particular memory. Remembering
the past helps us to re-examine our lives, make choices, and share personal
accomplishments. We often use external devices to help us remember
events big and small, and with advances in technology, we often record and make
plans using a variety of digital devices such as iPhones, Microsoft’s Outlook,
and even smart watches. We have the capability to store a lifetime of data with
these advanced technologies, and with the advent of Facebook, Twitter, “selfies”,
and blogs it has become routine for many people to document their lives on a
daily basis in a digital form, a practice that has been referred to as
“lifelogging.” The outcome of documenting activities digitally are human
digital memories (HDM), which have been defined as “a combination
of many types of media, audio, video, images, and many texts of textual content [1].
The concept of recording and then later having the ability
to review certain documents was first proposed by Dr. Vannevar Bush in 1945
when he described the “Memex”
(a combination of “memory” and index”) in an issue of Atlantic Monthly [2]. As described in the article,
a Memex was “a device in which an individual stores all his books, records, and
communications, and which is mechanized so that it may be consulted with
exceeding speed and flexibility. It is an enlarged intimate supplement to his
memory.”The device would look like
a desk where documents were either recorded via microfilm or photography.
 |
| From u-tx.net |
Since that time, many
similar devices have been developed, but a revolutionary advance was seen with
Microsoft’s SenseCam a wearable camera with a wide-angle lens and multiple
sensors, including an infrared sensor to detect the presence of other people.
The camera takes a photo every 30 seconds, resulting in up to 2,500 photos a
day and is capable of storing 30,000 images in total. Photos can then uploaded
to a computer and viewed later using a Microsoft application [3]. SenseCam was developed as a memory aid and there has been over 50 research institutions
that have used the device in a variety of studies involving memory and behavior [4]. Notably the SenseCam has shown promising results in studies where it
was used as memory aid for a child with anterograde amnesia [5] and with adult patients that were
suffering from amnesia [3]. Aside from the medical
purpose that a camera such as SenseCam
could potentially serve, “lifelogging” has become more socially acceptable as
we live in a digital age where Facebook posts and Twitter feeds are consumed
constantly, and “selfies”
are a regular occurrence at most events.
 |
| The SenseCam. From microsoft.com |
However, our memories and our experiences are made up of more
than static images. Our memories are composed of sensory information, such as
temperature and smells, and especially emotions and physiological signals. The
next step of HDM would involve going beyond just digital images, and instead
would include physiological information that has been captured with sensors.
This type of information could allow for a more vivid recall and potentially
could remind us how we felt at any point in time [4]. Wearable systems that
incorporate sensors which collect data for future review have been the focus of
many researchers. One example of this type of system is the “Physiological Sensor Suite (PSS),” which
collects electrocardiogram (ECG), electromyogram (EMG), electrooculogram (EOG)
and through-hair electroencephalogram (EEG) that is then sent wirelessly to a
data sensor [6]. Although the sensors of this
suite did not require a gel to be applied on the skin first, the most
effortless way to use sensors to record our daily activities would be smart
fabrics, such as “smart shirts” that combine textiles and wireless sensors
to monitor heart rate, angle of inclination, body temperature, and location [7]. Shirts of this type were originally
designed for a hospital setting as a noninvasive method to monitor patients,
but in the age of personal computing, technologies like this could potentially
be used outside of a hospital.
 |
| An example of a "smart shirt". From howstuffworks.com |
Of course, these
types of sensors that record physiological data such as heart rate or body
temperature would be essentially meaningless alone; it would be difficult to
reconstruct any memory, even a simple memory, based on physiological data since
it is so ambiguous. However, if physiological data were supplemented with more
data, such as a photo of the user, the location where the data was recorded,
and the temperature, these clues together may help to trigger certain memories
or parts of memories. One example of a system that works to combine data from a
variety of sources is the AffectAura, an emotional prosthetic
where data is collected from devices such as a microphone, Microsoft’s
Kinect, and a webcam to predict emotional states such as engagement,
valence, and arousal. Six participants were recorded over 4 days, and based on
the data collected, users were able to reconstruct stories about their days [8]. There are still multiple
challenges associated with creating accurate systems that could reveal
emotional aspects of a memory, especially when systems and devices are created
that go beyond only capturing photos. However, sensors will only become smaller
and smaller in the coming years, and as a society we have a great interest in
recording events for cultural reasons, so it is not unreasonable that one
point, most people will participate in the act of “lifelogging” and the
creation of HDM.
It is important to note though that physiological data or
even photos can only act as triggers to a memory, as there is no method to
actually capture an exact “memory” or a “thought.” Additionally, not only would
an HDM need to incorporate data from a variety of sensors and then correctly
corroborate and translate this data into meaningful information (a significant
challenge), but an entire lifetime of
memories would need be recorded for an accurate reflection and then a database
that is searchable would also be required.
If however we could accurately record and then disseminate
data that could compose a memory, does this documentation act as crutch?
Reminiscing and sharing personal stories with families and friends is a basic
human experience that acts as way to connect with others. If instead of
memorializing a lost relative through stories or laughing with friends over a
childhood experience, we could just push “play” on a device, how would that
change us? We already live in a society where any question can be answered with
a quick Google search on a phone requiring no discussion between two people, but
how would our interactions with others change if we could just “Google” how a
past experience played out or made us feel? (MIT professor Dr. Sherry Turkle has been studying how
technology impacts people for over 15 years and has written numerous articles
and books on the topics, and given this recent interesting TED talk on
the subject). Devices that use information from HDM are meant to help us with
the reminiscence process, but what if these devices are actually making us lose
that ability, or at the very least, fundamentally altering the memory process?
Additionally,
just as remembering is central to our existence, so is forgetting. Just because
at some point in the future we may be able to document a person’s entire
lifetime with a wealth of data, including physiological data, should we? Just
this year, researchers from the University of Basel discovered the musashi
protein, a protein that appears to inhibit molecules that stabilize synaptic
connections. These connections are important for the development of memories,
and based on this discovery, it appears that forgetting
is an active biological process. The biological processes behind
remembering and forgetting appear to work together, and forgetting is not just
a passive process [9]; we most likely forget for a
reason, even if more research is necessary to discover why.
References
1. Kelly L. The Information Retrieval
Challenge of Human Digital Memories. Proceedings of the 1st BCS IRSG Conference
on Future Directions in Information Access [Internet]. Swinton, UK, UK: British
Computer Society; 2007 [cited 2014 Aug 19]. p. 17–17. Available from:
http://dl.acm.org/citation.cfm?id=2227895.2227913
2.Bush V. As We May
Think. The Atlantic [Internet]. 1945
Jul [cited 2014 Aug 19]; Available from:
http://www.theatlantic.com/magazine/archive/1945/07/as-we-may-think/303881/
3. Hodges S, Williams L, Berry E, Izadi S, Srinivasan J, Butler A, Smyth G, Kapur N, Wood E. SenseCam: A
Retrospective Memory Aid. UbiComp 2006 4206:
177-193.
4. Fairclough, S.H., and Gilleade, K. (2014). Capturing Human Digital Memories for Assisting Memory Recall In Advances in Physiological Computing, S.H. Fairclough, and K. Gilleade, eds.(Springer London), pp. 211-234.
5. Pauly-Takacs K,
Moulin CJA, Estlin EJ. SenseCam as a rehabilitation tool in a child with
anterograde amnesia. Mem Hove Engl.
2011, 19(7): 705–12.
6. Matthews R,
McDonald NJ, Hervieux P, Turner PJ, Steindorf MA. A wearable physiological
sensor suite for unobtrusive monitoring of physiological and cognitive state. Conf Proc IEEE Eng Med Biol Soc 2007, 2007:
5276–81.
7. López G, Custodio
V, Moreno JI. LOBIN: E-textile and wireless-sensor-network-based platform for
healthcare monitoring in future hospital environments. IEEE Trans Inf Technol Biomed Publ 2010, 14(6): 1446–58.
8. Mcduff D, Karlson
A, Kapoor A, Roseway A, Czerwinski M. AffectAura: An Intelligent System forEmotional Memory.
9. Hadziselimovic N,
Vukojevic V, Peter F, Milnik A, Fastenrath M, Fenyves BG, et al. Forgetting Is
Regulated via Musashi-Mediated Translational Control of the Arp2/3 Complex. Cell, 2014, 156(6): 1153–66.
Want to cite this post?
Strong, K. (2014). “Lifelogging” and neurophysiological computing: Will we forget how to forget? The Neuroethics Blog. Retrieved on , from http://www.theneuroethicsblog.com/2014/08/lifelogging-and-neurophysiological.html
Comments
Post a Comment