USING SMARTPHONES AS PERSONAL MONITORING TOOLS FOR THE ACOUSTIC

ENVIRONMENT!

João Cordeiro!Álvaro Barbosa!

School of Arts - Portuguese Catholic University – C. R. Porto!CITAR - Research Centre for Science And Technology of The Arts (Porto)!Faculty of Creative Industries - University of Saint Joseph – Macau, China!

Outline!

•  Main goals!•  Problem statement!•  System design!•  Experimental procedure!•  Results!•  Conclusions!•  Future work!

Main goals!

Better qualityof life!

Achieve better soundscapes in

urban areas!

Increase awareness for the acoustic phenomenon!

Problem statement!

How to promote sound awareness? !

Education!Music > Sound !Sound walks!Sound design!

Public Politics!Campaigns !

Laws!Training!

Technology!Entertaining / Useful / Social!

Problem statement!Entertaining / Useful / Social MOBILE Technology!

!!Advantages!

!- People are keen to new technologies!!- Mostly mobile-based!!- Powerful computing processing!!- Feature several sensors (microphones, camera, accelerometer…)!!- Open to developers!!- …!

!Disadvantages!

!- Not rigorously calibrated!!- Not multitasking friendly!!- Power autonomy!!- Privacy issues !!!- Latency communication!!- …!

!

Problem statement!!!>> SOUND BASED <<!!!Audio Games!(ex: Murky Shooting)!!!

Entertaining / Useful / Social MOBILE Technology!

Problem statement!!!>> SOUND BASED <<!!!Audio Games!!!Sound Maps!(ex. Freesound uploads) !!!

Entertaining / Useful / Social MOBILE Technology!

Problem statement!!!>> SOUND BASED <<!!!Audio Games!!!Sound Maps!!!Sound-Based Social Networks!(ex. Hurly-Burly)!

Entertaining / Useful / Social MOBILE Technology!

Problem statement!

Can we use soundscapes to improve Online Social Networking?!(thus creating an engaging service based on sound)!

System design!

Hurly-Burly!(means: boisterous activity) !!

•  senses the sound environment surrounding each user!•  classifies it according to the sound sources!•  measures the sound pressure level!•  implements a online social network!•  distributes the sound metadata through the social network!•  keep a record of user’s soundscapes automatic metadata!

System design!

System design!

•  iOS platform!•  Senses the soundscapes using built

in microphone!

•  Classification: Music, Speech, Environmental Sounds, Silence!

•  Measures SPL – dB(A)Leq, 3s!

•  Continuous Monitoring (background)!

•  Online Social Network!

•  Movement Sensing!

System design!

System design!

•  MySQL database!•  PHP web interface!

•  Users’ soundscape logs!- timestamp !- soundscape type !- SPL!

•  Relational table (who is friends with who)!

System design!

•  Interactive timeline graph with users’ soundscapes and movement data!

•  Affords horizontal zoom in/out and timeline “traveling”!

System design!

Experimental procedure!

•  Test the system with 3 test groups!•  Total 23 users + 2 control devices!•  3 days each test!•  Surveys and data logs were collected!

Each group is a Social Network!with Cluster Coefficient = 1!

Results!

1  

10  

2  

1  

5  

1  

1   1  

1  

USJ   UCP   HB  

Portuguese   Philippine   Chinese   Israeli   Spanish   Brazilian   Czech  

Sample characterization | nationality !

Results!

Sample characterization | age distribution!

0  

1  

2  

3  

4  

19   20   21   22   23   24   25   26   27   28   29   30   31   32   33   34   35   36   37   38   39   40   41   42   43  

Num

ber  o

f  sub

jects  

Age  range  

USJ   UCP   HB  

Results!

Sample characterization | frequency of Social Media usage !

18  

2  

2  

1  

Everyday  

Twice  a  week  

Once  a  month  

I'm  no  sure  

Results!

Sample characterization | favourite way to access Social Media!

35%  

48%  

13%  

4%  

Mobile  devices  (smartphones,  iPods...)  

Laptops  

Tablets  

Desktops  

3  

3  

4  

7  

9  

13  

14  

17  

Who  is  with  me  

My  Loca[on  

What  music  Am  I  

Videos  

Audio  

Things  I  Like  

Textual  messages  

Photos  

Results!

Sample characterization | what users prefer to share on Social Media!

non-contextual cues !

Results!

Experiment characterization | users proximity with the device during exp.!

48%  52%  

Yes,  I  carried  the  device  always  with  me  

Most  of  the  [me,  but  not  always  

No,  We  were  separate  

Results!

Experiment characterization | devices used to run the application!

65%  

18%  

17%  

iPhone  

iPod  

iPad  

Results!

Experiment characterization | application performance per device!

0%  

10%  

20%  

30%  

40%  

50%  

60%  

70%  

80%  

90%  

100%  

iphone   ipod   ipad  

Major  Porblems  

Minor  Porblems  

Excelent  

Results!

User experience | what was the first reason to switch of the application!

1  

10  

3  

0  

0  

0  

All  the  above  

Low  ba_ery  

Privacy  issues  

Keeps  my  device  slow  

I  don't  likethe  red  bar  

I'm  not  sure  

Results!

User experience | audio clips v.s. soundscapes automatic metadata!

3  

7  

9  

4  

I  would  use  it,  I  think  it  increases  the  sense  of  realism  of  the  social  experience  

I  might  use  it  with  cau[on  

I  would  use  it  only  in  very  controlled  situa[ons.  

I  would  never  use  an  applica[on  like  that,  it  violates  my  privacy  

Results!

User experience | users’ knowledge of the term soundscape !

70%  

4%  

26%  

Yes  

No  

More  or  less  

Results!

User experience | hurly-burly systems as environmental gauges!

2  

7  

14  

I  doubt   I  guess  so   Yes,  definitely  

Results!

User experience | choices according the knowledge of soundscapes !

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100%  

I  know   More  or  Less  

I  Doubt  

Maybe  

Define_ly  

Results!

Soundscape history | Example 1 – two days with the same pattern!

Results!

Soundscape history | Example 2 – office at CITAR – 3 weeks log!

Results!

Soundscape history | Example 2a – detail – 10h to 2h (late meeting)!

Results!

Soundscape history | Example 3 – 1,5 days - Macau!

Conclusions!•  Mobile devices are a great opportunity for sensing the environment,

however, multitasking and power constraints may hinder this use!

•  Soundscapes provide an interesting description of people’s activity, which can be applied to many areas where this information is relevant, for example Social Networking!

•  Recording and sharing audio clips may hinder the use of sound in Social Networking, due to privacy issues. Classifying the sound with machine audition techniques represents a good option!

•  There’s a relation between knowing the concept of soundscape and acknowledging it as an important resource. Thus an increase of education on this field will lead to more conscientious citizens about their sound environment!

Future work!

•  Port the application for other platforms (Android)!

•  Enrich the study with more users!

•  Interleave the application with existent Online Social Networks (e.g. Facebook, Google +, Twitter…)!

•  Developing other technological applications that may lead to an increase of the sound awareness among non-experts!

•  …!

Questions?!

Thank you!!

João Cordeiro!Álvaro Barbosa!

School of Arts - Portuguese Catholic University – C. R. Porto!CITAR - Research Centre for Science And Technology of The Arts (Porto)!Faculty of Creative Industries - University of Saint Joseph – Macau, China!