A Distributed Architecture for the Monitoring and Analysis of Time Series Data

A Distributed Architecture for the Monitoring and Analysis of Time Series Data
Author :
Publisher : Ruairí O'Reilly
Total Pages :
Release :
ISBN-10 :
ISBN-13 :
Rating : 4/5 ( Downloads)

Book Synopsis A Distributed Architecture for the Monitoring and Analysis of Time Series Data by : Ruairi O'Reilly

Download or read book A Distributed Architecture for the Monitoring and Analysis of Time Series Data written by Ruairi O'Reilly and published by Ruairí O'Reilly. This book was released on with total page pages. Available in PDF, EPUB and Kindle. Book excerpt: Abstract It is estimated that the quantity of digital data being transferred, processed or stored at any one time currently stands at 4.4 zettabytes (4.4 × 270 bytes) and this figure is expected to have grown by a factor of 10 to 44 zettabytes by 2020 [1]. Exploiting this data is and will remain, a significant challenge. At present there is the capacity to store 33% of digital data in existence at any one time; by 2020 this capacity is expected to fall to 15%. These statistics suggest that, in the era of Big Data, the identification of important, exploitable data will need to be done in a timely manner. Systems for the monitoring and analysis of data, e.g. stock markets, smart grids and sensor networks, can be made up of massive numbers of individual components. These components can be geographically distributed yet may interact with one another via continuous data streams, which in turn may affect the state of the sender or receiver. This introduces a dynamic causality, which further complicates the overall system by introducing a temporal constraint that is difficult to accommodate. Practical approaches to realising the system described above have led to a multiplicity of analysis techniques, each of which concentrates on specific characteristics of the system being analysed and treats these characteristics as the dominant component affecting the results being sought. The multiplicity of analysis techniques introduces another layer of heterogeneity, that is heterogeneity of approach, partitioning the field to the extent that results from one domain are difficult to exploit in another. The question is asked can a generic solution for the monitoring and analysis of data that: accommodates temporal constraints; bridges the gap between expert knowledge and raw data; and enables data to be effectively interpreted and exploited in a transparent manner, be identified? The approach proposed in this dissertation acquires, analyses and processes data in a manner that is free of the constraints of any particular analysis technique, while at the same time facilitating these techniques where appropriate. Constraints are applied by defining a workflow based on the production, interpretation and consumption of data. This supports the application of different analysis techniques on the same raw data without the danger of incorporating hidden bias that may exist. To illustrate and to realise this approach a software platform has been created that allows for the transparent analysis of data, combining analysis techniques with a maintainable record of provenance so that independent third party analysis can be applied to verify any derived conclusions. In order to demonstrate these concepts, a complex real-world example involving the near real-time capturing and analysis of neurophysiological data from a neonatal intensive care unit (NICU) was chosen. A system was engineered to gather raw data, analyse that data using different analysis techniques, uncover information, incorporate that information into the system and curate the evolution of the discovered knowledge. The application domain was chosen for three reasons: firstly because it is complex and no comprehensive solution exists; secondly, it requires tight interaction with domain experts, thus requiring the handling of subjective knowledge and inference; and thirdly, given the dearth of neurophysiologists, there is a real-world need to provide a solution for this domain.


A Distributed Architecture for the Monitoring and Analysis of Time Series Data Related Books

A Distributed Architecture for the Monitoring and Analysis of Time Series Data
Language: en
Pages:
Authors: Ruairi O'Reilly
Categories: Computers
Type: BOOK - Published: - Publisher: Ruairí O'Reilly

GET EBOOK

Abstract It is estimated that the quantity of digital data being transferred, processed or stored at any one time currently stands at 4.4 zettabytes (4.4 × 270
Site Reliability Engineering
Language: en
Pages: 552
Authors: Niall Richard Murphy
Categories:
Type: BOOK - Published: 2016-03-23 - Publisher: "O'Reilly Media, Inc."

GET EBOOK

The overwhelming majority of a software system’s lifespan is spent in use, not in design or implementation. So, why does conventional wisdom insist that softw
Client-server monitoring architecture for distributed supercomputing analytics
Language: en
Pages:
Authors: Milos Bozovic
Categories:
Type: BOOK - Published: 2018 - Publisher:

GET EBOOK

Machine learning is revolutionizing many industries, so health-care as well. Since it requires massive computing, one of the possible solutions to achieve such
Data Management Technologies and Applications
Language: en
Pages: 168
Authors: Slimane Hammoudi
Categories: Computers
Type: BOOK - Published: 2020-07-29 - Publisher: Springer Nature

GET EBOOK

This book constitutes the thoroughly refereed proceedings of the 8th International Conference on Data Management Technologies and Applications, DATA 2019, held
Collaborative Computing: Networking, Applications and Worksharing
Language: en
Pages: 412
Authors: Honghao Gao
Categories:
Type: BOOK - Published: - Publisher: Springer Nature

GET EBOOK