刘旭东 cameran@hit.edu.cn Sunday, March 12, 2017 软件体系结构 12 云计算架构 刘旭东 cameran@hit.edu.cn Sunday, March 12, 2017

What Do Today’s Users Want? Accessibility Access from anywhere and from multiple devices Shareability Make sharing as easy as creating and saving Simplicity Easy-to-learn, easy-to-use Security Trust that data will not be lost or seen by unwanted parties

Issues Facing Web Developers 70% of Web Development Effort is “Muck”(体力活?) Data Centers Bandwidth / Power / Cooling Operations Staffing Scaling is Difficult and Expensive(艰难且昂贵!) Large Up-Front Investment Invest Ahead of Demand Load is Unpredictable 42% 基础架构维护 30% 应用程序维护 23% 应用程序投资 5% 基础架构投资

Acquisition cost is 10% of IT Spend software Acquisition cost is 10% of IT Spend hardware network facilities IT labor management tools power/cooling Operating cost is 90% of IT Spend support maintenance security disaster recovery backup Source: IDC

Technology Driving Factors Cheaper Hardware Broadband network Virtualization Web2.0 SaaS SOA …

Cloud Computing! Data stored on the cloud Software & services on the cloud - Access via web browser Based on standards and protocols - Linux, AJAX, LAMP, etc. Accessible from any device

主要内容 什么是云计算 云计算关键技术 云计算平台 云计算概念及架构 数据中心 云计算与网格计算 虚拟化 大规模分布式数据存储 海量数据处理 多租户 云计算平台 Amazon Google Azure Salesforce.com Hadoop Ezilla 参考书《虚拟化与云计算》

12.1 什么是云计算

What is Cloud Computing? “Cloud” refers to large Internet services like Google, Yahoo, etc that run on 10,000’s of machines, the emerging cloud abstracts infrastructure complexities of servers, applications, data, and heterogeneous platforms.(云是互联网服务，云是对复杂基础架构的抽象) “cloud computing” refers to services by these companies that let external customers rent computing cycles on their clusters (云计算是云提供商提供的租借计算服务，可使外部客户租借他们集群进行计算) Amazon EC2: virtual machines at 10¢/hour, billed hourly Amazon S3: storage at 15¢/GB/month

Essential Features of Cloud Computing Virtualization software, databases, Web servers, operating systems, storage and networking as virtual resources. On demand add and subtract processors, memory, network bandwidth, storage according to your needs

Architecture of Cloud Computing Services Resources managed at each layer Examples Google Apps, Facebook, YouTube Salesforce.com Software as a Service (SaaS) Business Applications Web Services, Multimedia Applications Runtimes, Operating System, Database Platform as a Service (PaaS), Google AppEngine, Microsoft Azure Platform Amazon EC2, Eucalyptus OpenNEbula Computing (VM), Storage Infrastructure Infrostructure as a Service (IaaS) Server, Network Data Centers Hardware

Cloud Delivery Models Software as a Service (SaaS)(软件即服务) Use provider’s applications over a network On-demand applications E.g. GMail, Microsoft Office Web Companions Platform as a Service (PaaS) (平台即服务) Deploy customer-created applications to a cloud On-demand application-hosting environment E.g. Google AppEngine, Salesforce.com, Windows Azure, Amazon Infrastructure as a Service (IaaS) (基础设施即服务) Rent processing, storage, network capacity, and other fundamental computing resources On-demand servers Amazon EC2, VMWare vCloud Cloud Software as a Service (SaaS). The capability provided to the consumer is to use the provider’s applications running on a cloud infrastructure and accessible from various client devices through a thin client interface such as a Web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings. Cloud Platform as a Service (PaaS). The capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created applications using programming languages and tools supported by the provider (e.g., java, python, .Net). The consumer does not manage or control the underlying cloud infrastructure, network, servers, operating systems, or storage, but the consumer has control over the deployed applications and possibly application hosting environment configurations. Cloud Infrastructure as a Service (IaaS). The capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, deployed applications, and possibly select networking components (e.g., firewalls, load balancers).

Cloud Delivery Models = Managed for You Standalone Servers IaaS PaaS SaaS Applications Runtimes Database Operating System Virtualization Server Storage Network

SaaS SaaS is a software delivery model in which software and associated data are centrally hosted on the cloud. SaaS is typically accessed by users using a thin client via a web browser.(是将软件和数据部署于云端的一种软件交付模式，客户通常通过浏览器使用软件) SaaS has become a common delivery model for most business applications, including accounting, invoicing, CRM, MIS, ERP, HRM, CM, etc. The vast majority of SaaS solutions are based on a multi-tenant architecture.(多租户架构) Example

PaaS PaaS is a category of cloud computing services that provide a computing platform and a solution stack as a service. (将计算平台及方案栈作为服务) In the PaaS model, cloud providers deliver a computing platform and/or solution stack typically including operating system, programming language execution environment, database, and web server.(提供商提供的平台包括操作系统、运行环境、数据库及Web服务器等) Various types of PaaS vendor offerings could be extensive and will include a total application hosting, development, testing, and deployment environment.(提供商提供程序开发、测试及部署环境) With some PaaS offers, the underlying compute and storage resources scale automatically to match application demand such that the cloud user does not have to allocate resources manually. (自动化资源分配)

IaaS In this most basic cloud service model, cloud providers offer computers – as physical or more often as virtual machines , raw (block) storage, firewalls, load balancers, and networks. IaaS providers supply these resources on demand from their large pools installed in data centers. (提供商按需提供虚拟机、存储、防火墙、负载均衡及网络等资源) In this model, it is the cloud user who is responsible for installing, patching and maintaining the operating systems and application software. (用户需自行维护操作系统和应用程序) Cloud providers typically bill IaaS services on a utility computing basis, that is, cost will reflect the amount of resources allocated and consumed. (提供商根据资源使用情况计费)

Levels of abstraction Different levels of abstraction Instruction Set VM: Amazon EC2 Application Level VM: Google AppEngine Similar to languages Higher level abstractions can be built on top of lower ones Lower-level, More flexibility, More management Not scalable by default Higher-level, Less flexibility, Less management Automatically scalable 17 EC2 Azure AppEngine Force.com

Example: Google Google云计算应用实例

Cloud Deployment Models Private/Community cloud (私有云) enterprise owned Or shared infrastructure for specific community Public cloud (公有云) Sold to the public, mega-scale infrastructure Hybrid cloud (混合云) composition of the two above clouds

Data Center(数据中心) Data centre is a facility used to house computer systems and associated components, such as telecommunications and storage systems. It generally includes redundant or backup power supplies, redundant data communications connections, environmental controls (e.g., air conditioning, fire suppression) and security devices. 数据中心是一整套复杂的设施。它不仅安装计算机系统和其它与之配套的设备(例如通信和存储系统),还包含冗余的电源、数据通信连接、环境控制设备(例如空调防火设备)以及各种安全装置。

Data Center 全球企业IT开支(IDC统计数据) $0B 50 100 150 200 250 300 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 New Server Spending Server Mgt and Admin Costs Power and Cooling Costs Source: IBM Corporate Strategy analysis of IDC data, Sept. 2007

Data Center 价格 地点 可能的定价原因 云环境下数据中心的经济性(管理/能耗) 项目 中型数据中心成本 特大型数据中心成本 比率 网络 $95 每 Mb/秒/月 $13 每 Mb/秒/月 7.1 存储 $2.20 每 GB/月 $0.40 每 GB/月 5.7 管理 每个管理员约管理140个服务器 每个管理员管理1000个服务器以上 价格 地点 可能的定价原因 3.6美分/千瓦时 爱达荷州 水力发电，没有长途输送 10.0美分/千瓦时 加州 电力在电网上长途输送，湾区的输送线有限，加州不允许煤电 18.0美分/千瓦时 夏威夷 发电的能源需要海运到岛上

Data Center 数据中心规模(2012年估计) Google服务器数量：约100万台(2012年) Amazon服务器数量：约45万台(2012年) Microsoft服务器数量：约21万台(2008年) Intel服务器数量：约10万台(2011年) Akamai服务器数量：约10.5万台(2012年3月) Facebook服务器数量：约6万台(2009年) Taobao CDN服务器数量：约4000台(2012年)

Grid Computing (网格计算) Grid computing is a computing model that provides the ability to perform higher throughput computing by taking advantage of many networked computers to model a virtual computer architecture that is able to distribute process execution across a parallel infrastructure. 网格计算：一种计算模型，通过将多台分布在网络上的计算机形成一个虚拟机，提供强大的、并行的计算能力

Grid Computing (网格计算) To solve large-scale computation problems (解决大规模的计算问题) breaking large data sets down into many smaller ones (将大数据集分解为若干个小的数据集) modeling a parallel division of labor (多台计算机进行并行的计算) Grids offer a way to solve Grand Challenge problems like (Grid为诸多挑战性的问题提供了解决途径) protein folding (蛋白质折叠) financial modelling (经济与金融建模) earthquake simulation (地震模拟) climate/weather modelling (气候/天气建模) astronomical searching (天体搜索)

Grid Computing & Cloud Computing 网格计算 异构资源 不同机构 虚拟组织 科学计算为主 高性能计算机 紧耦合问题 免费 标准化 科学界 云计算 同构资源 单一机构 虚拟机 数据处理为主 服务器/PC 松耦合问题 按量计费 尚无标准 商业社会

Grid Computing & Cloud Computing

12.2 云计算关键技术

Virtualization 虚拟化

Virtualization (虚拟化) Virtualization is the creation of a virtual (rather than actual) version of something, such as a hardware platform, operating system, storage device, or network resources. 虚拟化是为某些事物创建的虚拟版本，例如硬件平台、操作系统、存储设备或者网络资源等。 Types of Virtualization Infrastructure Virtualization (基础设施虚拟化) CPU、内存、设备/IO、网络 System Virtualization (系统虚拟化) 虚拟机，如Xen, VMware, VirtualBox Software Virtualization (应用虚拟化) 应用、语言，如JVM

16 Enterprise Edition licenses Virtualization (虚拟化) 虚拟化节省IT开支&提高资源利用率 SQL 8 servers 16 processors / licenses $100K $200K $300K $400K $500K $596K 16 Enterprise Edition licenses SA $600K 2 servers 4 processors / licenses $158K 4 licenses >70% cost reduction

Start “Single-step automatic recovery” Virtualization (虚拟化) 虚拟化加快系统部署及系统恢复 Configure hardware Install OS Configure OS Install backup agent Start “Single-step automatic recovery” Restore VM Power on VM Physical Virtual 40+ hrs < 4 hrs

Virtualization (虚拟化) 虚拟化实现高可用性(实时迁移Live Migration)

大规模分布式数据存储 Big Data

大规模分布式数据存储 Big data is a term applied to data sets whose size is beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed time.数据量巨大使得传统数据处理工具无法在可接受的时间内处理完毕) One current feature of big data is the difficulty working with it using relational databases and desktop statistics/visualization packages, requiring instead “massively parallel software running on tens, hundreds, or even thousands of servers”.(需要数以千计的服务器平行处理)

大规模分布式数据存储实例:GFS GFS架构 将文件划分为若干块（Chunk）存储 单一Master, 若干ChunkServer 通过冗余来提高可靠性 每个数据块至少在3个数据块服务器上冗余 直接读取ChunkServer数据

大规模分布式数据存储实例:GFS Master节点功能 存储元数据 文件系统目录管理与加锁 与ChunkServer进行周期性通信 发送指令，搜集状态，跟踪数据块的完好性 数据块创建、复制及负载均衡 对ChunkServer的空间使用和访问速度进行负载均衡，平滑数据存储和访问请求的负载 对数据块进行复制、分散到ChunkServer上 一旦数据块冗余数小于最低数，就发起复制操作 垃圾回收 在日志中记录删除操作，并将文件属性设为隐藏 缓慢地回收隐藏文件 与传统文件删除相比更简单、更安全 陈旧数据块删除 探测陈旧的数据块，并删除

大规模分布式数据存储实例:GFS GFS特点 采用中心服务器模式 不缓存数据 在用户态下实现 提供专用的访问接口 可以方便地增加Chunk Server Master掌握系统内所有Chunk Server的情况，方便进行负载均衡 不存在元数据的一致性问题 不缓存数据 GFS的文件操作大部分是流式读写，不存在大量的重复读写，使用Cache对性能提高不大 Chunk Server上的数据存取使用本地文件系统，如果某个Chunk读取频繁，文件系统具有Cache 从可行性看，Cache与实际数据的一致性维护也极其复杂 在用户态下实现 直接利用Chunk Server的文件系统存取Chunk，实现简单 用户态应用调试较为简单，利于开发 用户态的GFS不会影响Chunk Server的稳定性 提供专用的访问接口 未提供标准的POSIX访问接口 降低GFS的实现复杂度

大规模分布式数据存储实例:GFS GFS性能

Mass Data Processing 海量数据处理

Mass Data Processing(海量数据处理) 待处理数据量巨大（PB级），只有分布在成百上千个节点上并行计算才能在可接受的时间内完成. Index construction for Google Search Spam detection for Yahoo! Mail 如何进行并行分布式计算？ MapReduce is a programming model and an associated implementation for processing and generating large datasets that is amenable to a broad variety of real-world tasks. It was introduced by Jeffery Dean in 2004. MapReduce: Simplified Data Processing on Large Clusters, Communications of the ACM, vol. 51, no. 1 (2008), pp. 107-113 Jeffrey Dean and Sanjay Ghemawat.

MapReduce Applications @ Google: Index construction for Google Search Article clustering for Google News Statistical machine translation @ Yahoo!: “Web map” powering Yahoo! Search Spam detection for Yahoo! Mail @ Facebook: Data mining Ad optimization Spam detection

MapReduce Development Users implement interface of two primary methods（用户需要实现两个基本函数）: map (k1,v1) → list(k2,v2) reduce (k2,list(v2)) → list(v2) Map, written by the user, takes an input pair and produces a set of intermediate key/value pairs. The MapReduce library groups together all intermediate values associated with the same intermediate key I and passes them to the reduce function. (Map操作将输入转换为中间的KV对， 然后MapReduce库将同一个key的所有values传递给reduce函数) The reduce function, also written by the user, accepts an intermediate key I and a set of values for that key. It merges these values together to form a possibly smaller set of values. (reduce函数对传递过来的Values对进行合并处理) Draw an analogy to SQL, map can be visualized as group-by clause of an aggregate query.

MapReduce Example: Word Count Consider the problem of counting the number of occurrences of each word in a large collection of documents. http://www.cnblogs.com/xia520pi/archive/2012/05/16/2504205.html

MapReduce Example: Word Count The user would write code similar to the following pseudo-code.

MapReduce Example: Execution Overview

MapReduce Example: Word Count Step 1 1. The MapReduce library in the user program first splits the input files into M pieces of typically 16-64MB per piece (controllable by the user via an optional parameter). It then starts up many copies of the program on a cluster of machines. 2. One of the copies of the program—the master— is special. The rest are workers that are assigned work by the master. There are M map tasks and R reduce tasks to assign. The master picks idle workers and assigns each one a map task or a reduce task. 3. A worker who is assigned a map task reads the contents of the corresponding input split. It parses key/value pairs out of the input data and passes each pair to the user-defined map function. The intermediate key/value pairs produced by the map function are buffered in memory. 4. Periodically, the buffered pairs are written to local disk, partitioned into R regions by the partitioning function. The locations of these buffered pairs on the local disk are passed back to the master who is responsible for forwarding these locations to the reduce workers. Step 2-4

MapReduce Example: Word Count 5. When a reduce worker is notified by the master about these locations, it uses remote procedure calls to read the buffered data from the local disks of the map workers. When a reduce worker has read all intermediate data for its partition, it sorts it by the intermediate keys so that all occurrences of the same key are grouped together. The sorting is needed because typically many different keys map to the same reduce task. If the amount of intermediate data is too large to fit in memory, an external sort is used. 6. The reduce worker iterates over the sorted intermediate data and for each unique intermediate key encountered, it passes the key and the corresponding set of intermediate values to the user’s reduce function. The output of the reduce function is appended to a final output file for this reduce partition. Step 5 Step 6

MapReduce Statistics 对1000台计算机上的1TB数据进行排序仅需要68s 对4000台计算机上的1PB数据进行排序处理仅需要6小时2分钟（每次测试至少会损坏1块硬盘） 在08年1月份，Google MapReduce平均每天的数据处理量约20PB

Multi-tenant Architecture (MTA) 多租户架构

Multi-tenant Architecture (MTA) Multi-tenancy(多租户) refers to a principle in software architecture where a single instance of the software runs on a server, serving multiple client organizations (tenants). (单个应用实例服务过个客户组织) With a multitenant architecture, the applications are typically required to provide a high degree of customization to support each target organization's needs.(多租户架构中，每一个租户都对应一个个性化的配置) Because of the additional customization complexity and the need to maintain per-tenant metadata, multitenant applications require a larger development effort. (客户化的复杂性导致开发正本高) SaaS中的关键技术

Multi-tenant Architecture (MTA) What is Tenet (租户)?与用户的区别？ An Example: 基于SaaS的网上电影院订票系统 SaaS平台的管理人员是一类用户 登录网上订票系统订票的影迷是系统的另一类用户 租用该系统的电影院的是租户 电影院内部员工通过系统进行影讯发布和售票，他们是租户内的用户

4-Level Maturity Model of SaaS Tenant 1 Tenant 2 Tenant 3 (a)Ad-Hoc/Custom (b) Configurable (c) Configurable, Multi-Tenant-Efficient (d) Scalable, Configurable, Load Balancer From Tsai Wei-Tek

Level 1 - Ad-Hoc / Custom Each customer has its own customized version of the hosted application and runs its own instance of the application on the host's servers.

Level 2 - Configurable Greater program flexibility through configurable metadata, so that many customers can use separate instances of the same application code.

Level 3 - Configurable, Multi-Tenant-Efficient Adds multi-tenancy so that a single program instance serves all customers.

Level 4 - Scalable, Configurable, Multi-Tenant-Efficient Adds scalability through a multi-tier architecture supporting a load-balanced farm of identical application instances, running on a variable number of servers.

Resource Isolation Patterns of Data Tier in MTA Separate databases (SD) Each tenant has its own database Other resources (CPU, storage, code) shared Easy to extend data model and recover Expensive in maintaining many copies of DBs

Resource Isolation Patterns of Data Tier in MTA Shared database separate schemas (SDSS) Each tenant has its own database schemas Easy to implement and extend data model

Resource Isolation Patterns of Data Tier in MTA Shared database shared schemas (SDSHS) All tenants share the same schemas, easy to maintain Most complext due to security reasons Hard to restore after failures

12.3 主流云计算平台

Amazon云计算平台AWS IaaS PaaS SaaS Paying for What You Use 弹性计算云EC2 (IaaS) Small (Default) $0.10 per hour $0.125 per hour All Data Transfer $0.10 per GB 简单存储服务S3 (PaaS) $0.150 per GB – first 50 TB / month of storage used $0.100 per GB – all data transfer in $0.01 per 1,000 PUT, COPY, POST, or LIST requests PaaS 简单数据库服务Simple DB 简单队列服务SQS 弹性MapReduce 内容推送服务CloudFront SaaS 电子商务服务DevPay 灵活支付服务FPS 虛擬化的技術：Amazon EC2 Small (Default) $0.10 per hour $0.125 per hour All Data Transfer $0.10 per GB 儲存服務：Amazon S3 $0.150 per GB – first 50 TB / month of storage used $0.100 per GB – all data transfer in $0.01 per 1,000 PUT, COPY, POST, or LIST requests 觀念：Paying for What You Use Paying for What You Use

Google云计算平台 Google云计算平台技术架构 Google App Engine (PaaS) 文件存储，Google Distributed File System，GFS 并行数据处理MapReduce 分布式锁Chubby 结构化数据表BigTable Google App Engine (PaaS) 500MB of storage up to 5 million page views a month 10 applications per developer account Language: Python、Java Google Applications (SaaS) Google Docs Google Mail Google Calendar …

微软云计算平台Azure platform as service

Salesforce.com云计算平台

Salesforce.com云计算平台 Database Server: Oracle RAC EE, Dell, Sun Clustering: SunCluster Web Site and Application Server: Dell, Resin Search Server: Jakarta Lucene Storage Management: Hitachi Data Systems, Sun Backup Software: Veritas/RMAN Operating Systems Sun Solaris Redhat Linux

开源云计算平台Hadoop Open Source Apache Project Hadoop Core includes: Distributed File System - distributes data Map/Reduce - distributes application Written in Java Runs on Linux, Mac OS/X, Windows, and Solaris Commodity hardware Hadoop云计算系统 对应Google云计算系统 Hadoop HDFS Google GFS Hadoop MapReduce Google MapReduce Hadoop HBase Google Bigtable Hadoop ZooKeeper Google Chubby

Try It! 开源云计算集成平台Ezilla Ezilla (Easy La) helps users quickly and easily build their own private cloud! Ezilla integrates storage, networking, and computing resources under a cloud and allows cloud service providers to swiftly deploy cloud service environments that are built on a user-friendly, web-based interface. Integrated Open source resources Linux (开源OS: hosted machine OS & Virtual machine OS) OpenNEbula (类似EC2的开源实现) EyeOS (开源云操作系统) Xen (开源虚拟机) Try It! http://ezilla.info/

结束 2017年3月12日