SlideShare a Scribd company logo

Effective IoT System on Openstack

•
•
Takashi Kajinami
Takashi Kajinami

This document discusses building an effective IoT system on OpenStack. It describes key IoT use cases and requirements, such as high data volume, velocity, and variety. The proposed architecture uses OpenStack services like Nova, Neutron, Swift, and Ceilometer to provide scalable infrastructure, networking, storage, and monitoring for IoT workloads. The document outlines how OpenStack can support broker integration, device management, flexible data stores, external connectivity, and data federation to realize a full-featured IoT platform. Future work involves proof-of-concept testing of the integrated architecture.

1 of 39
Download to read offline
Copyright © 2015 NTT DATA Corporation October 27, 2015 Hiroshi Miura & Takashi Kajinami Effective IoT System on OpenStack
2Copyright © 2015 NTT DATA Corporation Disclaimer • Any product name, service name, software name and other marks are trademark or registered trademark of corresponding companies. • This presentation is in a purpose of providing the knowledge gained from our project, but no guarantee for its correctness. • A presenter and NTT Data Corporation provide information in as-is basis and have no responsiveness for results that you got according to information in this presentation material.
3Copyright © 2015 NTT DATA Corporation About us  Who are presenters? • Hiroshi Miura : Team Manager / OpenStack ATC for Swift / Linux Kernel Contributor / Samba contributor / and so on… • Takashi Kajinami : Platform engineer/consultant / OpenStack ATC for Swift  Working on OSS professional services sector  NTT DATA is ICT professional service firm supporting customers in world wide.
4Copyright © 2015 NTT DATA Corporation Team mission  OSS Professional service with OpenStack • Provides Cloud technology (OpenStack, Sheepdog, Docker,…) for customer • Realize automation of platform deployment and operations  Extend Application field for Cloud Technologies. • Build and provide IoT aware cloud platform
5Copyright © 2015 NTT DATA Corporation Agenda • Overview • What is IoT? • Use cases • IoT platform • Requirements • Why to use cloud • Our architecture plan • Effective IoT on OpenStack • How to build IoT platform on OpenStack • Future work
Copyright © 2015 NTT DATA Corporation 6 Overview about IoT
7Copyright © 2015 NTT DATA Corporation IoT: Internet of things Everything is connected to the Internet Internet Computers Smart devices Consumer electronics Cars Construction Machines IoT enables data gathering, controlling over the Internet
8Copyright © 2015 NTT DATA Corporation Big data analysis with IoT Gather data from anything, and make value through analyzing them
9Copyright © 2015 NTT DATA Corporation Use case #1 : Sensor network
10Copyright © 2015 NTT DATA Corporation Use case #2 : Industry 4.0
11Copyright © 2015 NTT DATA Corporation Use case #3: Connected Car / Auto drive vehicle The Connected Car, MWC 2013 by Ericson https://www.flickr.com/photos/ericsson_images/8511950335
Copyright © 2015 NTT DATA Corporation 12 How to realize IoT platform?
13Copyright © 2015 NTT DATA Corporation 3 Key features of IoT data Volume Velocity Variety > petabytes >10Gbps >10ktps >1M connections > 100 data formats > 100 analytic method
14Copyright © 2015 NTT DATA Corporation 2 Key requirements for IoT platform Volume >petabytes Velocity >10Gbps >10ktps >1M connections Variety >100 data formats >100 analytic method Scalable storage Scalable compute/network Flexible compute/storage/network Cloud is the best solution
15Copyright © 2015 NTT DATA Corporation Other requirements Compute Storage Network Security
16Copyright © 2015 NTT DATA Corporation Requirements for IoT infrastructure Compute • DevOps support • Continuous delivery • Automation • Automated deployment • Orchestration • Automated operation (ex. Auto-scaling, HA)
17Copyright © 2015 NTT DATA Corporation Requirements for IoT infrastructure Network • Protocol support • MQTT • Kafka • COAP • REST • QoS Control • Bandwidth • Priority (QoS0/1/2) • Latency • Throughput • Session & Authentication management • 3G/4G/5G network • Neighborhood communications
18Copyright © 2015 NTT DATA Corporation Requirements for IoT infrastructure Storage • Variety of data format • Record • Document • Binary • Cooperation with computing resources • Effectively process data in storage
19Copyright © 2015 NTT DATA Corporation • Multi-tenancy • For PaaS/IaaS • For big data processing platform • For data storage • Dashboard • Authentication/Authorization • Authentication of users • Authentication of devices • Authorization to connect • 3G/4G network accounting • Encryption • Data stored • Network communication Requirements for IoT infrastructure Security Manufactured security by krls krüg https://www.flickr.com/photos/kk/23390123
20Copyright © 2015 NTT DATA Corporation Cloud Authentication/Authorization IoT architecture in NTT DATA Scalable cloud infrastructure Cloud Monitoring Device Authentication/Authorization Gateway Data source Data source Data source Gateway Data source Data source Data source Internet Device Monitoring Analysis App Visual App Broker cache RDB NoSQL Object Storage Data Processing (bt, rt) Adaptor Receiver Converter&Accumulator L3Router,VPN PaaS/ Container Math Pkg connected car Sensor app BEMS HEMS
21Copyright © 2015 NTT DATA Corporation Analysis App Visual App Math Pkg connected car Sensor app BEMS HEMS Cloud Authentication/Authorization IoT architecture in NTT DATA Scalable cloud infrastructure Cloud Monitoring Device Authentication/Authorization Gateway Data source Data source Data source Gateway Data source Data source Data source Internet Device Monitoring Broker cache RDB NoSQL Object Storage Data Processing (bt, rt) Adaptor Receiver Converter&Accumulator L3Router,VPN PaaS/ Container
22Copyright © 2015 NTT DATA Corporation Data collections Gateway Data source Data source Data source Gateway Data source Data source Data source Internet L3Router,VPN
23Copyright © 2015 NTT DATA Corporation Integrated Platforms Cloud Authentication/Authorization Scalable cloud infrastructure Cloud Monitoring Device Authentication/Authorization Device Monitoring Gate way Data sourc e Data sourc e Data sourc e Gate way Data sourc e Data sourc e Data sourc e Inter net Applicati on Applicati on Broker cache RDB NoSQL Object Storage Data Processing (bt, rt) Adaptor Receiver Converter& Accumulator L3Router,VPN PaaS/ Container Applicati on Applicati on Applicati on Applicati on
24Copyright © 2015 NTT DATA Corporation Data integration core Broker cache RDB NoSQL Object Storage Data Processing (bt, rt) Adaptor Receiver Converter&Accumulator L3Router,VPN Analysis App Visual App PaaS/ Container Math Pkg connected car Sensor app BEMS HEMS Cloud Authentication/Authorization Scalable cloud infrastructure Cloud Monitoring Device Authentication/Authorization Device Monitoring
25Copyright © 2015 NTT DATA Corporation Analysis apps market places Analysis App Visual App PaaS/ Container Math Pkg connect ed car Sensor app BEMS HEMS Broker cache RDB NoSQL Object Storage Data Processing (bt, rt) Adaptor Receiver Converter& Accumulator L3Router,VPN
Copyright © 2015 NTT DATA Corporation 26 IoT system on OpenStack
27Copyright © 2015 NTT DATA Corporation Cloud Authentication/Authorization IoT architecture on OpenStack Scalable cloud infrastructure Cloud Monitoring Device Authentication/Authorization Gateway Data source Data source Data source Gateway Data source Data source Data source Internet Nova/Ironic Keystone Broker cache RDB NoSQL Object Storage Data Processing (bt, rt) Adaptor Receiver Converter&Accumulator Trove Sahara Swift L3Router,VPN Neutron Device Monitoring ceilometer PaaS/ Container Heat, Magnum Trove Trove Mature Testing Dev Analysis App Visual App Math Pkg connected car Sensor app BEMS HEMS
28Copyright © 2015 NTT DATA Corporation MATURITY LEVELS for related initiatives Keynote speech at OpenStack Tokyo Summit day 1 by Jonathan Bryce
29Copyright © 2015 NTT DATA Corporation What we can do for IoT 1. Broker support 2. Integrated device management 3. Monitoring integration 4. Management of data stores 5. Connectivity to external networks 6. Data federation and integration
30Copyright © 2015 NTT DATA Corporation What we can do for IoT 1. Broker support • Broker products • Mosquitto, Apache Kafka • IBM MessageSight • Broker service is different from SQS service(Zaqar)
31Copyright © 2015 NTT DATA Corporation What we can do for IoT 2. Integration of device management • Integrated management • Device management • Authentication and authorization for devices • Cloud management • Data management • Extensibility of Identification Livid Instruments by Brain jr https://www.flickr.com/photos/31245410@N07/8456344050
32Copyright © 2015 NTT DATA Corporation • Monitor resources inside and outside OpenStack • Virtual machines • Networks • Devices • Data Processing status • Machine learning clusters • No integrated solutions today • Ceilometer and Monasca? • ELK • Other tools? (Nagios, Ichinga, Zabbix, Hinemos, …) What we can do for IoT 3. Monitoring Photo by NEC Corporation of America with Creative Commons license.
33Copyright © 2015 NTT DATA Corporation What we can do for IoT 4. Management of data stores • Flexible management of variety of data stores • RDB: MySQL, PostgreSQL, … • NoSQL: mongoDB, CouchDB, … • KVS(cache): Redis, Couchbase, Cassandra, … • Object Storage • Supports for these data stores is now under dev/test • Construction of scalable cluster • Sharding cluster of mongodb • Redis cluster
34Copyright © 2015 NTT DATA Corporation What we can do for IoT 5. Connectivity to external network • Network traffic between internal network and external one is important. • Incoming network traffic is heavier than the other. • Network node should be bottle neck with conventional L3 architecture Compute Node Compute Node Compute Node Internet VM VM Network Node ■DVR Architecture Bottle neck External Network Internal Network
35Copyright © 2015 NTT DATA Corporation What we can do for IoT 6. Data federation • Sequential use of data over data stores and data processing platform • Move data between data stores • Process data in data stores • Store result of processing to data stores Broker Object Storage cache raw data (like raw image) NoSQL Move cached data to storage Store received data to cache Detect stored data and process Store result to another data store Data Processing processed data (like feature data) camera
36Copyright © 2015 NTT DATA Corporation What we can do for IoT 1. Broker support 2. Integrated device management 3. Monitoring integration 4. Management of data stores 5. Connectivity to external networks 6. Data federation and integration
37Copyright © 2015 NTT DATA Corporation Summery • IoT technologies enable us to gather huge data over the internet, and create new value with data analysis. • Flexibility and scalability are necessary feature to realize IoT platform, and cloud is the best answer for the requirements. • We put our IoT architecture on OpenStack, and summarized to-do list to make effective IoT platform on OpenStack.
38Copyright © 2015 NTT DATA Corporation Future work • PoC of IoT platform on cloud • Verification of each software has partly finished • Construction of whole architecture after testing each components • We will feedback that knowledge to the community.
Copyright © 2011 NTT DATA Corporation Copyright © 2015 NTT DATA Corporation

More Related Content

Effective IoT System on Openstack

  • 1. Copyright © 2015 NTT DATA Corporation October 27, 2015 Hiroshi Miura & Takashi Kajinami Effective IoT System on OpenStack
  • 2. 2Copyright © 2015 NTT DATA Corporation Disclaimer • Any product name, service name, software name and other marks are trademark or registered trademark of corresponding companies. • This presentation is in a purpose of providing the knowledge gained from our project, but no guarantee for its correctness. • A presenter and NTT Data Corporation provide information in as-is basis and have no responsiveness for results that you got according to information in this presentation material.
  • 3. 3Copyright © 2015 NTT DATA Corporation About us  Who are presenters? • Hiroshi Miura : Team Manager / OpenStack ATC for Swift / Linux Kernel Contributor / Samba contributor / and so on… • Takashi Kajinami : Platform engineer/consultant / OpenStack ATC for Swift  Working on OSS professional services sector  NTT DATA is ICT professional service firm supporting customers in world wide.
  • 4. 4Copyright © 2015 NTT DATA Corporation Team mission  OSS Professional service with OpenStack • Provides Cloud technology (OpenStack, Sheepdog, Docker,…) for customer • Realize automation of platform deployment and operations  Extend Application field for Cloud Technologies. • Build and provide IoT aware cloud platform
  • 5. 5Copyright © 2015 NTT DATA Corporation Agenda • Overview • What is IoT? • Use cases • IoT platform • Requirements • Why to use cloud • Our architecture plan • Effective IoT on OpenStack • How to build IoT platform on OpenStack • Future work
  • 6. Copyright © 2015 NTT DATA Corporation 6 Overview about IoT
  • 7. 7Copyright © 2015 NTT DATA Corporation IoT: Internet of things Everything is connected to the Internet Internet Computers Smart devices Consumer electronics Cars Construction Machines IoT enables data gathering, controlling over the Internet
  • 8. 8Copyright © 2015 NTT DATA Corporation Big data analysis with IoT Gather data from anything, and make value through analyzing them
  • 9. 9Copyright © 2015 NTT DATA Corporation Use case #1 : Sensor network
  • 10. 10Copyright © 2015 NTT DATA Corporation Use case #2 : Industry 4.0
  • 11. 11Copyright © 2015 NTT DATA Corporation Use case #3: Connected Car / Auto drive vehicle The Connected Car, MWC 2013 by Ericson https://www.flickr.com/photos/ericsson_images/8511950335
  • 12. Copyright © 2015 NTT DATA Corporation 12 How to realize IoT platform?
  • 13. 13Copyright © 2015 NTT DATA Corporation 3 Key features of IoT data Volume Velocity Variety > petabytes >10Gbps >10ktps >1M connections > 100 data formats > 100 analytic method
  • 14. 14Copyright © 2015 NTT DATA Corporation 2 Key requirements for IoT platform Volume >petabytes Velocity >10Gbps >10ktps >1M connections Variety >100 data formats >100 analytic method Scalable storage Scalable compute/network Flexible compute/storage/network Cloud is the best solution
  • 15. 15Copyright © 2015 NTT DATA Corporation Other requirements Compute Storage Network Security
  • 16. 16Copyright © 2015 NTT DATA Corporation Requirements for IoT infrastructure Compute • DevOps support • Continuous delivery • Automation • Automated deployment • Orchestration • Automated operation (ex. Auto-scaling, HA)
  • 17. 17Copyright © 2015 NTT DATA Corporation Requirements for IoT infrastructure Network • Protocol support • MQTT • Kafka • COAP • REST • QoS Control • Bandwidth • Priority (QoS0/1/2) • Latency • Throughput • Session & Authentication management • 3G/4G/5G network • Neighborhood communications
  • 18. 18Copyright © 2015 NTT DATA Corporation Requirements for IoT infrastructure Storage • Variety of data format • Record • Document • Binary • Cooperation with computing resources • Effectively process data in storage
  • 19. 19Copyright © 2015 NTT DATA Corporation • Multi-tenancy • For PaaS/IaaS • For big data processing platform • For data storage • Dashboard • Authentication/Authorization • Authentication of users • Authentication of devices • Authorization to connect • 3G/4G network accounting • Encryption • Data stored • Network communication Requirements for IoT infrastructure Security Manufactured security by krls krüg https://www.flickr.com/photos/kk/23390123
  • 20. 20Copyright © 2015 NTT DATA Corporation Cloud Authentication/Authorization IoT architecture in NTT DATA Scalable cloud infrastructure Cloud Monitoring Device Authentication/Authorization Gateway Data source Data source Data source Gateway Data source Data source Data source Internet Device Monitoring Analysis App Visual App Broker cache RDB NoSQL Object Storage Data Processing (bt, rt) Adaptor Receiver Converter&Accumulator L3Router,VPN PaaS/ Container Math Pkg connected car Sensor app BEMS HEMS
  • 21. 21Copyright © 2015 NTT DATA Corporation Analysis App Visual App Math Pkg connected car Sensor app BEMS HEMS Cloud Authentication/Authorization IoT architecture in NTT DATA Scalable cloud infrastructure Cloud Monitoring Device Authentication/Authorization Gateway Data source Data source Data source Gateway Data source Data source Data source Internet Device Monitoring Broker cache RDB NoSQL Object Storage Data Processing (bt, rt) Adaptor Receiver Converter&Accumulator L3Router,VPN PaaS/ Container
  • 22. 22Copyright © 2015 NTT DATA Corporation Data collections Gateway Data source Data source Data source Gateway Data source Data source Data source Internet L3Router,VPN
  • 23. 23Copyright © 2015 NTT DATA Corporation Integrated Platforms Cloud Authentication/Authorization Scalable cloud infrastructure Cloud Monitoring Device Authentication/Authorization Device Monitoring Gate way Data sourc e Data sourc e Data sourc e Gate way Data sourc e Data sourc e Data sourc e Inter net Applicati on Applicati on Broker cache RDB NoSQL Object Storage Data Processing (bt, rt) Adaptor Receiver Converter& Accumulator L3Router,VPN PaaS/ Container Applicati on Applicati on Applicati on Applicati on
  • 24. 24Copyright © 2015 NTT DATA Corporation Data integration core Broker cache RDB NoSQL Object Storage Data Processing (bt, rt) Adaptor Receiver Converter&Accumulator L3Router,VPN Analysis App Visual App PaaS/ Container Math Pkg connected car Sensor app BEMS HEMS Cloud Authentication/Authorization Scalable cloud infrastructure Cloud Monitoring Device Authentication/Authorization Device Monitoring
  • 25. 25Copyright © 2015 NTT DATA Corporation Analysis apps market places Analysis App Visual App PaaS/ Container Math Pkg connect ed car Sensor app BEMS HEMS Broker cache RDB NoSQL Object Storage Data Processing (bt, rt) Adaptor Receiver Converter& Accumulator L3Router,VPN
  • 26. Copyright © 2015 NTT DATA Corporation 26 IoT system on OpenStack
  • 27. 27Copyright © 2015 NTT DATA Corporation Cloud Authentication/Authorization IoT architecture on OpenStack Scalable cloud infrastructure Cloud Monitoring Device Authentication/Authorization Gateway Data source Data source Data source Gateway Data source Data source Data source Internet Nova/Ironic Keystone Broker cache RDB NoSQL Object Storage Data Processing (bt, rt) Adaptor Receiver Converter&Accumulator Trove Sahara Swift L3Router,VPN Neutron Device Monitoring ceilometer PaaS/ Container Heat, Magnum Trove Trove Mature Testing Dev Analysis App Visual App Math Pkg connected car Sensor app BEMS HEMS
  • 28. 28Copyright © 2015 NTT DATA Corporation MATURITY LEVELS for related initiatives Keynote speech at OpenStack Tokyo Summit day 1 by Jonathan Bryce
  • 29. 29Copyright © 2015 NTT DATA Corporation What we can do for IoT 1. Broker support 2. Integrated device management 3. Monitoring integration 4. Management of data stores 5. Connectivity to external networks 6. Data federation and integration
  • 30. 30Copyright © 2015 NTT DATA Corporation What we can do for IoT 1. Broker support • Broker products • Mosquitto, Apache Kafka • IBM MessageSight • Broker service is different from SQS service(Zaqar)
  • 31. 31Copyright © 2015 NTT DATA Corporation What we can do for IoT 2. Integration of device management • Integrated management • Device management • Authentication and authorization for devices • Cloud management • Data management • Extensibility of Identification Livid Instruments by Brain jr https://www.flickr.com/photos/31245410@N07/8456344050
  • 32. 32Copyright © 2015 NTT DATA Corporation • Monitor resources inside and outside OpenStack • Virtual machines • Networks • Devices • Data Processing status • Machine learning clusters • No integrated solutions today • Ceilometer and Monasca? • ELK • Other tools? (Nagios, Ichinga, Zabbix, Hinemos, …) What we can do for IoT 3. Monitoring Photo by NEC Corporation of America with Creative Commons license.
  • 33. 33Copyright © 2015 NTT DATA Corporation What we can do for IoT 4. Management of data stores • Flexible management of variety of data stores • RDB: MySQL, PostgreSQL, … • NoSQL: mongoDB, CouchDB, … • KVS(cache): Redis, Couchbase, Cassandra, … • Object Storage • Supports for these data stores is now under dev/test • Construction of scalable cluster • Sharding cluster of mongodb • Redis cluster
  • 34. 34Copyright © 2015 NTT DATA Corporation What we can do for IoT 5. Connectivity to external network • Network traffic between internal network and external one is important. • Incoming network traffic is heavier than the other. • Network node should be bottle neck with conventional L3 architecture Compute Node Compute Node Compute Node Internet VM VM Network Node ■DVR Architecture Bottle neck External Network Internal Network
  • 35. 35Copyright © 2015 NTT DATA Corporation What we can do for IoT 6. Data federation • Sequential use of data over data stores and data processing platform • Move data between data stores • Process data in data stores • Store result of processing to data stores Broker Object Storage cache raw data (like raw image) NoSQL Move cached data to storage Store received data to cache Detect stored data and process Store result to another data store Data Processing processed data (like feature data) camera
  • 36. 36Copyright © 2015 NTT DATA Corporation What we can do for IoT 1. Broker support 2. Integrated device management 3. Monitoring integration 4. Management of data stores 5. Connectivity to external networks 6. Data federation and integration
  • 37. 37Copyright © 2015 NTT DATA Corporation Summery • IoT technologies enable us to gather huge data over the internet, and create new value with data analysis. • Flexibility and scalability are necessary feature to realize IoT platform, and cloud is the best answer for the requirements. • We put our IoT architecture on OpenStack, and summarized to-do list to make effective IoT platform on OpenStack.
  • 38. 38Copyright © 2015 NTT DATA Corporation Future work • PoC of IoT platform on cloud • Verification of each software has partly finished • Construction of whole architecture after testing each components • We will feedback that knowledge to the community.
  • 39. Copyright © 2011 NTT DATA Corporation Copyright © 2015 NTT DATA Corporation