Enabling Conversational Interaction with Mobile UI using Large Language Models | Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

Large language models (LLMs) have recently been applied in software engineering to perform tasks such as translating code between programming languages, generating code from natural language, and autocompleting code as it is being written. When used ...

https://dl.acm.org/doi/10.1145/3544548.3580895

Introduction

Conversation For Mobile UI Tasks

Prompting LLMs for Mobile UI Tasks

Screen Representation

• View Hierarchy Properties
安卓界面的View Hierachy数据非常详细，如果直接作为输入会超过一些大模型的token limit。我们提取了一个和UI元素文本描述相关的子集：
Class: 元素的类型
Text: 元素内可见的文本内容
Resource_id: 表明元素相关资源的文本标识器
Content_desc: 辅助描述元素作用的内容

• Class Mapping
提取出的Class属性将会转变为对应的Html标签。

Classes	Html tag
TextView	<p>
button-related class	<button>
image-related class	<img>
EDITTEXT	<input>
the others	<div>

• Text, Resource_Id, and Content Description
我们会将text属性插入Html开始和结束的tag之间；其次是resource_id 他包含了三个实体，包名，资源类型，资源名字。其中，资源的名字是对元素功能的额外描述，所以我们将其做为class标识符的值插入到html中，因为其中经常会包含下划线，我们还会将_转化为空格；最后，我们将content description作为alt标识符的值插入到html中。

• Numeric Indexes for Referencing
我们会按照深度优先搜索view hierachy tree时的递增顺序，给每个元素的id标识符赋值，方便之后的各种指令通过id标识符选中UI元素。

Chain-of-Thought Prompting

Prompt Structure

Feasibility Experiments

Discussions And Future Work