Spring semester 2023

Human-Computer Interaction Seminar

This is the main seminar of the HCI group of the Human-IST Institute. The topics will change each semester and will be proposed by Human-IST members. Each participant will be supervised by one Human-IST member.

With the advent of autonomous cars, mobile devices and conversational agents, the question of the interaction with digital devices in everyday life is becoming more and more relevant. The aim of the HCI seminar is to look at this question over several specific contexts and expose students to state-of-the art research in Human-Computer Interaction.

Through a set of topics for the participants to choose from, the different research fields studied within the Human-IST institute will be reviewed and discussed. Again, each specific topic is proposed by a member of the Human-IST team who will be available, each time needed, during the semester to follow the work of the student selecting it.

Prof. Denis Lalanne

Dr Julien Nembrini (contact person)

The introductory lecture will be held on Thursday February 23 2023 10h45 in presence in PER21 A420. It will also be possible to attend online. Please contact Dr Julien Nembrini julien.nembrini@unifr.ch, with copy to Prof. Denis Lalanne denis.lalanne@unifr.ch, if you wish to attend.

Semester topics

Human-AI teaming for an augmented intelligence

Research and development in the field of artificial intelligence has so far focused mainly on technical aspects. It aims to mimic human performances. Augmented intelligence, on the other hand, takes a human-centered approach. It aims to support more efficient task performance by linking human and artificial intelligence. The goal of this seminar topic is to review research works on the topic of Human-AI teaming looking at dimensions such as ethics, sustainability, efficiency and regulation.

References:

Auernhammer, J. (2020). Human-centered AI: The role of Human-centered Design Research in the development of AI.

National Academies of Sciences, Engineering, and Medicine. "Human-AI Teaming: State-of-the-Art and Research Needs." (2021).

Reference person :

  • Dr Simon Ruffieux
  • Prof Denis Lalanne

The role of AI in Computer-Aided Diagnosis

Machine learning applications are already ubiquitous: conversational agents that sound almost aware are a hot topic, meanwhile do not even notice anymore our pictures getting auto-tagged on social media. And yet, so little of our lives are truly impacted by these technologies. From autonomous driving to medical care, anywhere trust is critical, having an intelligent human in the loop remains an absolute necessity. Why? Where is the threshold between "reliable", like a tool, and "trustworthy", like an intelligence? And what are the roles of the humans and AIs involved? To contribute to this discussion, it is paramount to first understand the limits and challenges of the current technologies and state of the art. This article highlights what makes it so hard to achieve "superhuman performance" in computer aided diagnosis for cancer detection (expert radiologists score a 0.76 in AUC for comparison), but the real question that needs answering: who bears the consequence of a false negative?

Challenges: this is a technical and clinical result that does not directly discusses the roles of human and computer. Single-mindedly improving performance without understanding its significance and applicability seldom leads to wider adoption. The student is required to contribute original, critical thinking, in how such a tool can be utilized by an expert physician, why should or should not be used, what are the consequences and responsibilities, and propose a safe way forward to expand the adoption of such methods.

References

Giuseppe Cuccu, Christophe Broillet, Carolin Reischauer, Harriet Thöny, and Philippe Cudré-Mauroux. “Typhon: Parallel Transfer on Heterogeneous Datasets for Cancer Detection in Computer-Aided Diagnosis.” In 2022 IEEE International Conference on Big Data, BigData, 2022, Osaka (JP), December 2022. https://exascale.info/assets/pdf/cuccu2022bigdata.pdf

Reference person :

  • Dr Giuseppe Cuccu

Companion interfaces to record individual energy behaviour

Smartphone-based analysis of personal mobility: technological challenges and privacy concerns

Smartphones are packed with sensors that can be used for the most disparate tasks. One of such tasks is the analysis of a person's mobility patterns and used modes of transportation. This information can be exploited to improve the user experience with contextual information, for example by activating contextual settings on the phone based on the modality in use; or to provide suggestions about how to modify one's patterns to promote healthy and sustainable mobility and help saving money.

The choice of the sensors to be used for this type of analysis, however, is not trivial. Indeed, the use of more sensitive data (e.g., precise location) or a high number of sensors, can provide a more precise analysis of mobility, but raises issues for the user, such as security and privacy concerns. Thus, the set of sensors or alternative methods providing a good compromise between accuracy and user satisfaction or willingness to adopt is researched in this work. This will be performed with the analysis of existing literature from the technical and social perspectives, as well as with the design of a user experiment.

References

Kaptchuk, G., Goldstein, D. G., Hargittai, E., Hofman, J. M., & Redmiles, E. M. (2022). How good is good enough? quantifying the impact of benefits, accuracy, and privacy on willingness to adopt covid-19 decision aids. Digital Threats: Research and Practice (DTRAP), 3(3), 1-18.

Hasan, R. A., Irshaid, H., Alhomaidat, F., Lee, S., & Oh, J. S. (2022). Transportation mode detection by using smartphones and smartwatches with machine learning. KSCE Journal of Civil Engineering, 26(8), 3578-3589.

Reference person :

  • Dr Moreno Colombo
  • Dr Julien Nembrini

Exploring the Use of Deep Learning Document Object Detection for Accessibility

Recent advances in the field of deep learning (DL), such as automatic text generation from images [1], has significantly improved the accessibility of computer systems for people with visual impairments (PVI). Moreover, advances in document object detection [4] made possible to improve the accessibility of PDF documents (i.e. converted as document images) or scanned document images. In that sense, such documents can benefit from labels detected to transcode a more accessible version [2] or enhance the navigability. However, DL‑based systems can provide non-perfect results [3], and little is known about how PVI deal with the results of such systems.

References

[1] Cole Gleason, Amy Pavel, Emma McCamey, Christina Low, Patrick Carrington, Kris M Kitani, and Jeffrey P Bigham. 2020. Twitter A11y: A Browser Extension to Make Twitter Images Accessible. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–12. Retrieved from https://doi.org/10.1145/3313831.3376728

[2] Simon Harper, Sean Bechhofer, and Darren Lunn. 2006. SADIe: Transcoding based on CSS. Eighth International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS 2006 2006, January: 259–260. https://doi.org/10.1145/1168987.1169044

[3] Haley MacLeod, Cynthia L Bennett, Meredith Ringel Morris, and Edward Cutrell. 2017. Understanding Blind People’s Experiences with Computer-Generated Captions of Social Media Images. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17), 5988–5999. https://doi.org/10.1145/3025453.3025814

[4] Zejiang Shen, Ruochen Zhang, Melissa Dell, Benjamin Charles Germain Lee, Jacob Carlson, and Weining Li. 2021. LayoutParser: A Unified Toolkit for Deep Learning Based Document Image Analysis. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 12821 LNCS, Dl: 131–146. https://doi.org/10.1007/978-3-030-86549-8_9

Reference person :

  • Maximiliano Jeanneret

Mental models: Toward a feeling of control for programming robots

The growing versatility of the market requires automated production systems to be easily and rapidly reprogrammed by the shop floor workers to meet dynamic production needs. This implies new types of interfaces that empower workers to not only acquire the skills for efficient and safe programming, but also to develop their self-confidence in autonomously using the technology. To bring this feeling of control, the user needs to develop a good understanding of how the robot works.

In order to develop empowering interface, we therefore need to better understand the critical information required by the user to improve his/her understanding of how the robot works, that is, his/her mental model of the robot's behaviour. Adapting the interface accordingly should enable the user to adapt his behaviour during his interaction with the machine to solve the different tasks, therefore improving his autonomy and developing a feeling of control over the machine.

Problematic: Which information and how much comprehension is needed for the end-user to feel in control when programming a robot?

Keywords:

End-User Robot Programming, Human-Robot Interaction, End-User Programming

References:

A Survey on End-User Robot Programming, Ajaykumar et al., ACM Computing Surveys, 2021 (section 6.2.3, https://arxiv.org/pdf/2105.01757.pdf)

The role of a mental model in learning to operate a device, Kieras and Bovair, Cognitive Science, 1984 (https://www.sciencedirect.com/science/article/abs/pii/S0364021384800038)

The role of mental model and shared grounds in human-robot interaction, Hwang et al., IEEE International Workshop on Robot and Human Interactive Communication, 2005 (https://ieeexplore.ieee.org/abstract/document/1513849)

Reference person :

  • Charly Blanc

Explainable artificial Intelligence for air traffic control

Due to the ongoing increase in air traffic density, performing air traffic control tasks is becoming more and more complex. In dense traffic situations, artificial intelligence (AI)-driven suggestions have the potential to improve safety and delays, and decrease the controller’s cognitive workload.

However, using ML-generated suggestions proves useful only if they can be understood, assessed and trusted by humans. The models behind these suggestions are complex and are not meant to be interpreted by humans. A user-centered approach is required in order to guarantee successful human-AI teaming, for example by using eXplainable AI (XAI). These new technologies need to comply with the new: rights and obligations have been introduced by the European Parliament in order to guarantee the safe, ethic, and legal aspects of the use of ML-driven automation.

This literature review will focus on new methods aimed at improving the human-AI teaming comprising the air traffic controller and AI-driven systems.

References:

Stephanidis, C., Salvendy, G., Antona, M., Chen, J. Y., Dong, J., Duffy, V. G., Fang, X., Fidopiastis, C., Fragomeni, G., Fu, L. P., & others. (2019). Seven HCI grand challenges. International Journal of Human–Computer Interaction, 35(14), 1229–1269.

Kistan, T., Gardi, A., & Sabatini, R. (2018). Machine Learning and Cognitive Ergonomics in Air Traffic Management: Recent Developments and Considerations for Certification. Aerospace, 5, 103.

Xu, W., Dainoff, M., Ge, L., & Gao, Z. (2023). Transitioning to Human Interaction with AI Systems: New Challenges and Opportunities for HCI Professionals to Enable Human-Centered AI. International Journal of Human-Computer Interaction, 39, 494–518.

Reference person :

  • Raphael Tuor

Work to be done

Students will be asked to :

  • Conduct an in-depth review of the state-of-the-art of their chosen topic
  • Discuss their findings with their respective tutor
  • Write a 4-pages article summarizing their review work
  • Present their article to the seminar participants

Interested computer science students are invited to participate by expressing their interest on two specific topics (preferred and secondary choices) among the ones presented on the topics presented above.

Learning Outcomes

At the end of the seminar, students will know how to do a bibliographic research and be exercised in writing a scientific article. Further, they will build a general knowledge on the field of HCI and its current techniques and trends, as well as an in-depth understanding on their chosen topic.

Registration

Attend the introductory session and express your interest about both your chosen topics with a short text (3-5 sentences) to Dr Julien Nembrini, mentioning (1) your preferred topic, (2) your second-preferred topic. Each reference person will then contact you if you are chosen for participating to the seminar. Others will receive a notification email.

emails : julien.nembrini@unifr.ch and denis.lalanne@unifr.ch

Registration Process

  1. Participate to the first introductory session on Thursday February 23 2023 10h45 in presence in PER21 A420.
  2. Express your interest for a specific topic as mentioned above before March 5 (first come first served).
  3. Wait for confirmation of your participation.

Seminar Process

In addition to the first session, there will only be three plenary sessions during the semester, which consist in participants presenting their work. These sessions will be in presence and will happen on Thursdays at 10h45. Reference persons will organize additional bilateral meetings during the semester.

Calendar (may be subject to changes)

  • February 23: initial seminar presentation
  • March 5: deadline for sending preferred topics
  • March 30: 1st presentation (state of the art)
  • April 27: 1st draft due
  • May 4: first draft presentation
  • May 25: 2nd draft due
  • June 1st: final presentations
  • June 15: final draft due

The seminar process is as follows:

  1. Select state-of-the-art references relevant to the chosen thematic, synthesize these references to structure the research field, discuss and refine your approach with your topic reference person.
  2. Present the structure developed to the other participants of the seminar for the intermediate presentation.
  3. Synthesize the selected bibliographic references in a written 4 pages article, authored in LaTeX following ACM SIGCHI format.
  4. Discuss and refine your article with your topic reference person.
  5. Present your article in one of the final presentation sessions (end of semester)

Evaluation

The evaluation will be conducted by the topic reference person and the person responsible for the seminar.

The evaluation will be based on the quality of :

  • Your written article: readability, argumentation, English
  • Your review work: reference selection and field structure
  • Your final presentation
  • Your initial draft
  • Your review of a colleague's first draft

If each step described below is not formally marked, each contribute towards producing a good final paper (75% of the grade)

Guidelines

First presentation

  • Introduce your theme
  • Your selection of 3 papers (the most relevant among all your readings according to your theme). Don't forget to give author names, journal and date!
  • Present for each paper:
    • Theme: How does the paper fit in the larger picture of your theme?
    • Main contribution: how does the paper contribute to the field?
    • Strengths and weaknesses of the proposed approach
    • Outlook: what research opportunities it opens?
  • Presentation time 10'.

First draft

  • your draft should :
    • present a research field with relevant publications
    • state a research question and an hypothesis
    • propose an experiment to test the hypothesis, and discuss its expected results
  • provide an (almost) complete draft. Enough content will allow reviewers to give meaningful feedback
  • polish your English grammar and orthography
  • you MUST use the latex template https://www.overleaf.com/latex/templates/association-for-computing-machinery-acm-sigchi- proceedings-template/nhjwrrczcyys
  • maximum 4 pages excluding references (do not hesitate to concentrate on some sections and leaving others in bullet point style)

Review

  • rephrase the content in 1 short paragraph
  • list the positive points
  • list the negative points, provide examples
  • propose improvements
  • do not insist on orthograph/grammar unless it is especially disturbing

Provide your review either as comments in the pdf or as a separate document.

Second presentation

  • your presentation should include:
    • Introduction to the topic
    • Research questions/hypotheses
    • General structure of your research field (literature review outcome)
    • User experiment proposal
  • include details (example papers, shared methods/approaches, experiments conducted by others, etc)
  • include full references (preferably not at the end)
  • present your review of fellow student's first draft (see guidelines above)
  • experiment with your presentation style
  • Presentation time 8 + 2 (review)

Second draft

  • combine feedback from your supervisor and from your fellow student

Final presentation

  • your presentation should include:
    • Introduction to the topic
    • Research questions/hypotheses
    • General structure of your research field (literature review outcome)
    • User experiment proposal
  • include details (example papers, shared methods/approaches, experiments conducted by others, etc)
  • include full references (preferably not at the end)
  • timing 10'

Final draft

  • integrate all feedback received and finalize your paper. this final version will be published on the website (habe a look here for previous examples)

Date: Spring 2023