16_与大象共舞——持续交付高质量的产品

© FreeWheel 2011 Dances with Elephants Build High Quality Product with Continuous Delivery DANG Zhengfa July, 2011 FreeWheel Confidential 22 Ad Networks All Key Distributors The Premium Video EcoSystem Connected TV’s, devices, platforms SERVING THE VIDEO REVOLUTION. 2 Confidential FreeWheel Clients 3 Confidential Challenges • Technic – Common to all Internet companies • UI: sophisticated workflow, Enormous validations • Ad Serving: High performance, high availability • Player SDK: Diversified platforms (flash, IOS, HTML5, android, Silverlight…) • Business – Special challenges for FreeWheel • Giant clients with huge impacts • Certification required: SAS70, MRC, IAB, and etc • Strict SLA and high penalty • Slow pace clients, but quick reaction required 4 Confidential 5 Confidential Dances with Elephants 6 Confidential 7 1. Mind Your Step: Define Features 2. Continuous Integration: Build Software 3. Managed Deployment 4. Continuous Testing in Production Confidential 1. Mind Your Step: Features Database 8 Confidential 1. Mind Your Step: Scrum V #1 Sprint Timeline (5 weeks) 1st Week 2nd Week 3rd Week 4th Week Tech Design & Implementation Integration Testing Staging CP0 Req Clarification Tech design Implementation Tech Design Implementation Testing on Staging Release to Prod. Begin req design 1st Week 2nd Week Overlapped four weeks Debating and Lock down the items Quality Assurance Testing on staging Release C2-2. Eng Hard commitments C3. PM Finish Details design Elaborate req and discussion Integration Testing on Staging 3rd Week 4th Week Last Week Last Week V #2 Sprint Timeline (5 weeks) Quality Assurance Code Freezing IC3: Code Freezing IC1: Design/ review took place or have been booked Eng planning IC 0: Eng plan done IC2: Test cases review C5. Staging Rel.C6. Prod. Rel. Req clarification Tech design Implementation C1. PM Rank Infrastructure rank C2-1. Eng Soft Commitments Prioritization Estimation NOTE: 30% ENG resources reserved for refactoring & emergency! FreeWheel Calendar Confidential 1. Mind Your Step • Limited Resources • Don’t be afraid to share your plans with clients • Pre-release noes, release note • Keep Schedule • Manage clients’ expectation • Things can’t be perfect at once • Beta 10 Confidential 2. Continuous Integration • Centralized Integration Server • Unified tools / libraries • Coding style, syntax check • Triggered by check-in, or periodically • Integrated with automated testing • Email Notification • Automated Testing • Frameworks • Units / E2E • Quick / Full Mode • Test Cases Prioritization 11 Confidential Player Simulator Run Data AdServer Lua Output Log Request Response Callbacks Response Base Regression Result 2. Continuous Integration: AdServer 12 Confidential 3. Deployment: Staging • Staging Environment 13 Simplified, fully OperationalSimplified, fully Operational Confidential 3. Deployment: Staging • Deployment rehearsal before production release • Identify bugs in “real” environment • Staging Environment • Mini production using virtualization tech • Real traffic: • Replay • Production-level monitoring • It DOES help! • AdServer crashed because of special cookies 14 Confidential 3. Deployment: Phase Rollout • Upgrade apps in multiple phases • Predefined groups, data centers 15 • Pros • Less risky • Natural fallback • Cons • Complexity • Compatibility test Confidential 3. Deployment: Automation • Deployment Automation • Staging ✘ Phase Rollout ✘ Releases = ? • Operation Infrastructure Project • Validation automation • Deployment, instead of features • Quick validation • Intuitive results 16 Confidential 4. Continuous Testing in Production System monitors • Disk / CPU / IO / Network • App Alerts / Errors External monitors • Gomez They are not enough! Real user simulation in production: • Dedicated client pages • Browser simulator to generate designed events • Player �� AdServer �� Reporting • Check final reporting results 17 Confidential 4. Continuous Testing in Production 18 Confidential 19 Confidential Q & A Email: zfdang@freewheel.tv 关注我们：t.baidu-tech.com 资料下载和详细介绍：infoq.com/cn/zones/baidu-salon InfoQ 策划·组织·实施 关注我们：weibo.com/infoqchina “畅想•交流•争鸣•聚会”是百度技术沙龙的宗旨。 百度技术沙龙是由百度与InfoQ中文站定期组织的线下技术交流活动。目 的是让中高端技术人员有一个相对自由的思想交流和交友沟通的的平台。主要分讲师分享和OpenSpace两个关键环节，每期 只关注一个焦点话题。 讲师分享和现场Q&A让大家了解百度和其他知名网站技术支持的先进实践经验，OpenSpace环节是百度技术沙龙主题的升华 和展开，提供一个自由交流的平台。针对当期主题，参与者人人都可以发起话题，展开讨论。