手势识别在自然交互中的应用

手势识别在自然交互中的应用 华南理工大学嵌入式系统 设计 领导形象设计圆作业设计ao工艺污水处理厂设计附属工程施工组织设计清扫机器人结构设计 论文 《智能机器人》课程实验报告 手势识别在自然交互中的应用 学 院 计算机科学与 工程 路基工程安全技术交底工程项目施工成本控制工程量增项单年度零星工程技术标正投影法基本原理 学院 专 业 计算机应用技术 组 员 徐庸辉 学 号 201120112011 学生导师 罗荣华 指导教师 肖南峰教授 提交日期 2012年7月1日 to West rear, to County Shen Liqun report guerrillas breakout Hou of action situation, boat via Zhejiang wuxing daughter town Shi, six people has a NI surname "spies" (enemy) midway tuogu landing tipsters, away from small town three in Xu, was day Elves Li Taishan Department captured, five people all killed, broken corpse sank Yu River in the. On January 18, 1943, when the Japanese army militiamen arrested 53 people in Wuzhen, transferred to the tomb of the former Japanese military police command (in the "Tai" Jiang Yuan), the 28th in Shenyang North of jade bang massacred them. Hu Maosheng knife wounds woke up in the middle of them, crawling out from the dead, in the tomb of Gao Changhai residents, saved by bing. Such as Hu Maosheng, Gao Changhai is a witness to this tragedy. (B) in memory of "massacre" Xu Youyong yan tomb is the copper law. Copper originally was called Luo yan Tomb, because Eastern Han dynasty distinguished prose poems my father yan Tan bogey was buried in the town's water. Has a long history here, cultural atmosphere, and there are many old houses, and left many people with lofty ideals. Business flourished here, is the hub of four townships of agricultural and sideline products, wine culture has a long history. The outskirts of the town has left, "Huang Chi", originally Yue water hides 智能机器人课程实验报告 目录 1. 实验要求......................................................................................................- 1 - 2. 设计平台......................................................................................................- 1 - 3. 实验原理......................................................................................................- 1 - 3.1. 手势识别现状及分类........................................................................... - 1 - 3.1.1. 静态手势识别................................................................................- 1 - 3.1.2. 动态手势识别................................................................................- 1 - 3.2. 场景建模方法....................................................................................... - 2 - 3.3. 自然交互与虚拟键盘........................................................................... - 3 - 4. 朴素手势识别 分析 定性数据统计分析pdf销售业绩分析模板建筑结构震害分析销售进度分析表京东商城竞争战略分析 .....................................................................................- 4 - 4.1. 轮廓关键点........................................................................................... - 4 - 4.2. 手势区域同心圆定位........................................................................... - 4 - 5. 实验设计......................................................................................................- 5 - 5.1. 背景消除 流程 快递问题件怎么处理流程河南自建厂房流程下载关于规范招聘需求审批流程制作流程表下载邮件下载流程设计 设计............................................................................... - 5 - 5.2. 手势识别流程设计............................................................................... - 6 - 5.3. 虚拟键盘设计....................................................................................... - 7 - 6. 实验结果......................................................................................................- 7 - 6.1. 手指数目识别结果............................................................................... - 7 - 6.2. 虚拟键盘输入结果............................................................................... - 9 - 7. 实验心得....................................................................................................- 10 - I 智能机器人课程实验报告 1. 实验要求 与自然语言相同，手势语言也能表达复杂信息，这样手势识别就可以进行人机交互，而不存在信息传达的困难，本文的目的是设计一个朴素的，无需分类的手势识别方法，并利用手势识别来实现基于自然交互原则的虚拟键盘。这样可以讲手势语言转换为自然语言。 2. 设计平台 操作系统为Windows XP，用VS2010 + QT + opencv作为开发平台。 3. 实验原理 3.1. 手势识别现状及分类 手势识别根据不同的识别对象，识别方法可分为静态手势识别和动态手势识别。目前流行算法的大致流程分为手势分割，手势建模，特征匹配。 图3-1 手势识别处理流程图 3.1.1. 静态手势识别 静态手势识别是针对单帧的手势图片做识别。 在手势分割的过程中主要使用肤色模型(色彩聚类特征)，轮廓边缘(canny检测)等; 在做手势建模时主要使用一些图像属性(轮廓，图像矩，特征，直方图); 特征匹配时使用模板匹配法，神经网络法等。 3.1.2. 动态手势识别 相比与静态手势识别，动态手势识别是针对手势图片序列来分析手势的含义。所以动态手势识别的手势分割涉及到运动区域检测，场景建模等。 在做手势分割时经常用到的运动跟踪方法有差值图像分割, camshaft, 卡尔曼跟踪预测, 背景剪除法。 , 在做手势建模时主要使用一些运动序列(计算运动光流, 抽取运动轨迹)来估算。 特征匹配时使用类似隐马尔科夫模型法、动态时间规划法等的一些方法。 3.2. 场景建模方法 运动目标检测是指在序列图像中检测出变化区域并将运动目标从背景图像中提取出来。通常情况下，目标分类、跟踪和行为理解等后处理过程仅仅考虑图像中对应于运动目标的像素区域，因此运动目标的正确检测与分割对于后期处理非常重要然而，由于场景的动态变化，如天气、光照、阴影及杂乱背景干扰等的影响，使得运动目标的检测与分割变得相当困难。 图3-2 帧差分处理效果对比图 根据摄像头是否保持静止，运动检测分为静态背景和运运动目标检测是指在序列图像中检测出变化区域并将运动目标从背景图像中提取出来。通常情况下，目标分类、跟踪和行为理解等后处理过程仅仅考虑图像中对应于运动目标的像素区域，因此运动目标的正确检测与分割对于后期处理非常重要然而，由于场景的动态变化，如天气、光照、阴影及杂乱背景干扰等的影响，使得运动目标的检测与分割变得相当困难。 根据摄像头是否保持静止，运动检测分为静态背景和运动背景两类。大多数视频监控系统是摄像头固定的，因此静态背景下运动目标检测算法受到广泛关注，常用的方法有帧差法、光流法、背景减除法，高斯背景模型等。 其中高斯背景模型效果比较好，但是由于我们的目的是利用朴素手势识别方法来完成虚拟键盘的设计，为了保证流畅的用户体验，减少图像分割在背景消除中的开销，这里我们选取最简单的帧差法。 图3-3 改进的背景消除方法效果图 houses, and left many people with lofty ideals. Business flourished here, is the hub of four townships of agricultural and simy father yan Tan bogey was buried in the town's water. Has a long history here, cultural atmosphere, and there are many old poems ouyong yan tomb is the copper law. Copper originally was called Luo yan Tomb, because Eastern Han dynasty distinguished proseai residents, saved by bing. Such as Hu Maosheng, Gao Changhai is a witness to this tragedy. (B) in memory of "massacre" Xu Yhanghbang massacred them. Hu Maosheng knife wounds woke up in the middle of them, crawling out from the dead, in the tomb of Gao C rred to the tomb of the former Japanese military police command (in the "Tai" Jiang Yuan), the 28th in Shenyang North of jaderansfed, broken corpse sank Yu River in the. On January 18, 1943, when the Japanese army militiamen arrested 53 people in Wuzhen, ttuogu landing tipsters, away from small town three in Xu, was day Elves Li Taishan Department captured, five people all killeShi, six people has a NI surname "spies" (enemy) midway to West rear, to County Shen Liqun report guerrillas breakout Hou of action situation, boat via Zhejiang wuxing daughter towne products, wine culture has a long history. The outskirts of the town has left, "Huang Chi", originally Yue water hidesdelin- 2 - 智能机器人课程实验报告 帧差法是最为常用的运动目标检测和分割方法之一，基本原理就是在图像序列相邻两帧或三帧间采用基于像素的时间差分通过闭值化来提取出图像中的运动区域。首先，将相邻帧图像对应像素值相减得到差分图像，然后对差分图像二值化，在环境亮度变化不大的情况下，如果对应像素值变化小于事先确定的阂值时，可以认为此处为背景像素:如果图像区域的像素值变化很大，可以认为这是由于图像中运动物体引起的，将这些区域标记为前景像素，利用标记的像素区域可以确定运动目标在图像中的位置。 由于相邻两帧间的时间间隔非常短，用前一帧图像作为当前帧的背景模型具有较好的实时性，其背景不积累，且更新速度快、算法简单、计算量小。算法的不足在于对环境噪声较为敏感，闽值的选择相当关键，选择过低不足以抑制图像中的噪声，过高则忽略了图像中有用的变化。对于比较大的、颜色一致的运动目标，有可能在目标内部产生空洞，无法完整地提取运动目标。 3.3. 自然交互与虚拟键盘 人机交互、人机互动(英文:Human–Computer Interaction[1]或Human–Machine Interaction，简称HCI或HMI)，是一门研究系统与用户之间的互动关系的学问。系统可以是各种各样的机器，也可以是计算机化的系统和软件。人机交互界面通常是指用户可见的部分。用户通过人机交互界面与系统交流，并进行操作。小如收音机的播放按键，大至飞机上的仪表板、或是发电厂的控制室。 人机交互界面的设计要包含用户对系统的理解(即心智模型)，那是为了系统的可用性或者用户友好性。 图3-4 人机手势交互效果图 在过去的20多年间，键盘和鼠标一直是人机对话的主要方式。不过科技在进步，也许在不久的将来，陪伴我们多年的键盘和鼠标就会消失在我们的视线中。虚拟激光键盘，是一种大小与小型移动电话相仿的虚拟键盘，让用户能像操作普通键盘一样轻易地打出文章或电子邮件、可方便地进行电子邮件收发、文字处理及电子表格制作，用户可以将 笔记 哲学笔记pdf明清笔记pdf政法笔记下载课堂笔记下载生物化学笔记PDF 本电脑和桌面机留在家里了。虚拟键盘的适用性技术对用户手指运动加以研究，对键盘击打动作进行解码和记录。由于虚拟键盘是光投照所形成的影像，不使用时会完全消失。 - 3 - 图3-5虚拟激光键盘 本文的目的是利用手势识别设计一个有利于人机自然交互的虚拟数字输入键盘。 4. 朴素手势识别分析 分割出场景中的手势轮廓后，我们通过一些朴素的方法对手势轮廓进行多边形逼近，然后提取关键点，然后对手势轮廓区域内不同半径圆环上的关键点的数量和位置做统计分析，从而识别出手势。 4.1. 轮廓关键点 轮廓的关键点是:轮廓上包含曲线信息比较多的点。关键点是轮廓顶点的子集。 使用cvFindDominantPoints函数来获取轮廓上的关键点，该函数返回的结果一个包含 关键点在轮廓顶点中索引 的序列(具体内容可查阅opencv参考手册)。 4.2. 手势区域同心圆定位 从手势轮廓的区域中心画出的如干个同心圆会与手指相交，连接圆心与关键点，然后计算每个同心圆与手指的交点，并统计每个同心圆上交点出现的位置和数量，然后分析出手指的数目。 如果想统计是那几只手指，或是想还原出具体的手势，可以对这些交点在每个同心圆上出现的位置和数量以及各交点之间的间隔做聚类分析或学习，这样能很好的达到更精确的识别目的。 实验中得到的关键点和同心圆如下图: 图4-1朴素手势识别效果图 delinhouses, and left many people with lofty ideals. Business flourished here, is the hub of four townships of agricultural and simy father yan Tan bogey was buried in the town's water. Has a long history here, cultural atmosphere, and there are many old poems ouyong yan tomb is the copper law. Copper originally was called Luo yan Tomb, because Eastern Han dynasty distinguished proseai residents, saved by bing. Such as Hu Maosheng, Gao Changhai is a witness to this tragedy. (B) in memory of "massacre" Xu Yhanghbang massacred them. Hu Maosheng knife wounds woke up in the middle of them, crawling out from the dead, in the tomb of Gao C rred to the tomb of the former Japanese military police command (in the "Tai" Jiang Yuan), the 28th in Shenyang North of jaderansfed, broken corpse sank Yu River in the. On January 18, 1943, when the Japanese army militiamen arrested 53 people in Wuzhen, ttuogu landing tipsters, away from small town three in Xu, was day Elves Li Taishan Department captured, five people all killeShi, six people has a NI surname "spies" (enemy) midway to West rear, to County Shen Liqun report guerrillas breakout Hou of action situation, boat via Zhejiang wuxing daughter towne products, wine culture has a long history. The outskirts of the town has left, "Huang Chi", originally Yue water hides- 4 - 智能机器人课程实验报告 从图中可以看出关键点的分布并不是按照期望的那样排列的，有很多杂点，我们通过多级同心圆的过滤分析，可以逐渐消除这些杂点。 5. 实验设计 为了减少由于手势轮廓提取带来的错误关键点并让用户得到比较好的虚拟键盘体验效果，在实验中我们选择比较简单的背景作为实验对象测试。 5.1. 背景消除流程设计 背景消除流程图: 输入图像 高斯滤波 帧差分 二值化前景图 更新背景 形态学滤波 输出前景 图5-1 背景消除流程图 关键函数: /******************************************************* * pFrame -初始图像帧 * pFrImg -前景图像 * pBkImg -背景图像 * pFrameMat -前景图像的中间矩阵 * pBkMat -背景图像的中间矩阵 ********************************************************/ CvtColor(pFrame, pFrImg, CV_BGR2GRAY); Convert(pFrImg, pFrameMat); //高斯滤波先，以平滑图像 Smooth(pFrameMat, pFrameMat, CV_GAUSSIAN, 3, 0, 0); //当前帧跟背景图相减 AbsDiff(pFrameMat, pBkMat, pFrMat); //二值化前景图 Threshold(pFrMat, pFrImg, 30, 255.0, CV_THRESH_BINARY); //进行形态学滤波，去掉噪音 Erode(pFrImg, pFrImg, 0, 1); Dilate(pFrImg, pFrImg, 0, 1); - 5 - //更新背景 RunningAvg(pFrameMat, pBkMat, 0.0001, 0); //将背景转化为图像格式，用以显示 Convert(pBkMat, pBkImg); 5.2. 手势识别流程设计 手势识别流程图: 手势轮廓提取 轮廓多边形逼近 查找多边形关键点 统计关键点数量 统计关键点位置 综合分析识别 图5-2 手势识别流程图 关键函数: /******************************************************* * srcImg -原始图像 * storage -临时存储 * contour -图像轮廓 * storageApp -临时存储 * storageDom -临时存储 * mode -轮廓查找计算方法 * CV_CHAIN_APPROX_SIMPLE -轮廓选择模式，压缩水平，垂直或斜的部分 * CV_POLY_APPROX_DP -多边形逼近参数 * CV_DOMINANT_IPAN -关键点查找参数 * dmin -最短距离(参考4.1部分) *dmax -最长距离(参考4.1部分) *dn -相邻距离(参考4.1部分) *maxAngle -最大角度(参考4.1部分) ********************************************************/ //查找轮廓 FindContours( srcImg, storage, &contour, sizeof(CvContour),mode, CV_CHAIN_APPROX_SIMPLE, Point(0,0)); //多边形逼近 ApproxPoly(contour,sizeof(CvContour),storageApp,CV_POLY_APPROX_DP,20,0); poems ouyong yan tomb is the copper law. Copper originally was called Luo yan Tomb, because Eastern Han dynasty distinguished proseai residents, saved by bing. Such as Hu Maosheng, Gao Changhai is a witness to this tragedy. (B) in memory of "massacre" Xu Yhanghbang massacred them. Hu Maosheng knife wounds woke up in the middle of them, crawling out from the dead, in the tomb of Gao C rred to the tomb of the former Japanese military police command (in the "Tai" Jiang Yuan), the 28th in Shenyang North of jaderansfed, broken corpse sank Yu River in the. On January 18, 1943, when the Japanese army militiamen arrested 53 people in Wuzhen, ttuogu landing tipsters, away from small town three in Xu, was day Elves Li Taishan Department captured, five people all killeShi, six people has a NI surname "spies" (enemy) midway to West rear, to County Shen Liqun report guerrillas breakout Hou of action situation, boat via Zhejiang wuxing daughter towne products, wine culture has a long history. The outskirts of the town has left, "Huang Chi", originally Yue water hidesdelinhouses, and left many people with lofty ideals. Business flourished here, is the hub of four townships of agricultural and simy father yan Tan bogey was buried in the town's water. Has a long history here, cultural atmosphere, and there are many old - 6 - 智能机器人课程实验报告 //查找关键点 FindDominantPoints(contour,storageDom,CV_DOMINANT_IPAN,dmin,dmax,dn,maxAngle) 5.3. 虚拟键盘设计 虚拟键盘处理流程图: 状态位初始化 输入关键点 检索最优关键点 修改状态位 检查当前状态 撤销焦点 获得焦点 选中焦点内容 输出 图5-3 虚拟键盘处理流程图 6. 实验结果 6.1. 手指数目识别结果 实验步骤: 步骤一:链接摄像头(分辨率大于640*480)到电脑上 步骤二:打开可执行文件，调整摄像头 步骤三:点击开始，查看界面gesture img extract窗口 步骤四:等待gesture img extract窗口基本没有噪声点，此时背景消除模型已基本 建立，可以开始识别 界面窗口: Virtual keyboard:虚拟数字键盘输入窗口 Original image:原始视频图像显示窗口 gesture img extract:手势识别处理结果显示图 InPut Num:虚拟数字键盘输入结果显示窗口 Fingures Num:手指数目识别结果显示窗口 - 7 - 图6-1手指数目识别效果图1 图6-2 手指数目识别效果图2 实验效果: 从6-1,6-2中可以看到，在手势基本稳定后，Fingers Num 标签处能正确显示手指的数目。有时可能由于手势变化过快，或由于长时间手势停留，会导致背景建模 e products, wine culture has a long history. The outskirts of the town has left, "Huang Chi", originally Yue water hidesdelinhouses, and left many people with lofty ideals. Business flourished here, is the hub of four townships of agricultural and simy father yan Tan bogey was buried in the town's water. Has a long history here, cultural atmosphere, and there are many old poems ouyong yan tomb is the copper law. Copper originally was called Luo yan Tomb, because Eastern Han dynasty distinguished proseai residents, saved by bing. Such as Hu Maosheng, Gao Changhai is a witness to this tragedy. (B) in memory of "massacre" Xu Yhanghbang massacred them. Hu Maosheng knife wounds woke up in the middle of them, crawling out from the dead, in the tomb of Gao C rred to the tomb of the former Japanese military police command (in the "Tai" Jiang Yuan), the 28th in Shenyang North of jaderansfed, broken corpse sank Yu River in the. On January 18, 1943, when the Japanese army militiamen arrested 53 people in Wuzhen, ttuogu landing tipsters, away from small town three in Xu, was day Elves Li Taishan Department captured, five people all killeShi, six people has a NI surname "spies" (enemy) midway to West rear, to County Shen Liqun report guerrillas breakout Hou of action situation, boat via Zhejiang wuxing daughter town- 8 - 智能机器人课程实验报告 错误，出现错误识别的现象(如图 6-3)，但是，总体的识别效果还是比较理想的。 6.2. 虚拟键盘输入结果 图6-3虚拟键盘数字键获得焦点图 图6-4 虚拟键盘数字键被选中图 - 9 - 实验效果: 从6-3,6-4中可以看到，在手指位置变化的过程中，虚拟键盘窗口区域能够准确的反应手势的输入情况。 图6-3 展示的是手势在数字键”5”上悬停的结果，此时数字键”5”获得焦点。 图6-4 展示的是手势选中数字键”5”的结果，此时Input Num 窗口区域中的数字从 ”*110*258#” 变为 “*110*258#5”，表示选中数字键”5”。 更多的实验效果可以查看实验录像。 7. 实验心得 通过对手势识别的研究与虚拟数字键盘的设计开发，理解了一般手势识别的处理过程，深入理解并掌握手势识别的原理及实际应用，同时对人机自然交互也有了新的认识。 从实验的效果来看，在不分类训练的情况下，朴素手势识别方法可以准确的识别手指的数目，能及时捕获关键点在虚拟键盘上的位置，并且可以实时的处理手势在虚拟键盘上的输入。从实验中可以看出手势识别对于人机自然交互有很好的帮助作用，能够改善传统的输入方式，给人们带来新的体验。 但是，可以从中知道手势识别在该应用中，仍存在某些缺陷:例如手势语义分类理解问题，虚拟键盘的多点触控问题等。目前的算法还只能识别出手指数目并对虚拟键盘单点做检查并做输入识别，但是很多情况下要求程序理解手势的语义而不仅仅是手指数目;同时为了增强人机交互的体验效果，单点的触控效果显然是不够的，用户需要更多的手势触控点。 e products, wine culture has a long history. The outskirts of the town has left, "Huang Chi", originally Yue water hidesdelinhouses, and left many people with lofty ideals. Business flourished here, is the hub of four townships of agricultural and simy father yan Tan bogey was buried in the town's water. Has a long history here, cultural atmosphere, and there are many old poems ouyong yan tomb is the copper law. Copper originally was called Luo yan Tomb, because Eastern Han dynasty distinguished proseai residents, saved by bing. Such as Hu Maosheng, Gao Changhai is a witness to this tragedy. (B) in memory of "massacre" Xu Yhanghbang massacred them. Hu Maosheng knife wounds woke up in the middle of them, crawling out from the dead, in the tomb of Gao C rred to the tomb of the former Japanese military police command (in the "Tai" Jiang Yuan), the 28th in Shenyang North of jaderansfed, broken corpse sank Yu River in the. On January 18, 1943, when the Japanese army militiamen arrested 53 people in Wuzhen, ttuogu landing tipsters, away from small town three in Xu, was day Elves Li Taishan Department captured, five people all killeShi, six people has a NI surname "spies" (enemy) midway to West rear, to County Shen Liqun report guerrillas breakout Hou of action situation, boat via Zhejiang wuxing daughter town- 10 -