Cespe UnB

Editorial Office:
Management:
R. S. Oyarzabal

Technical Support:
D. H. Diaz
M. A. Gomez
W. Abrahão
G. Oliveira

Publisher by
Knobook Pub






216/168=1.29


1.17

Exploratory study of the ELK stack for meteorological observation system data analysis

doi: 10.6062/jcis.2017.08.03.0130

(Free PDF)

Authors

E. S. Almeida, I. Koga, M. A. A. Santana, P. L. O. Guimarães, L. M. Sugawara and T. Eklin

Abstract

Different kinds of sensors compose a meteorological observation system that measures meteorological variables. Sensors can collect data for a long period of time in a high sampling frequency. Some meteorological parameters can be determined by making measurements that ranges from a few seconds to annual measurements which depends on the kind of equipment and application needs. In this scenario, data management is not a trivial task due to heterogeneity, large amount of data and also to the usage of proprietary software for data gathering and handling. We used a data acquisition system (datalogger) to collect and store data from a thermo-baro-hygrometer, and a pyranometer, which were calibrated previously in the laboratory. This paper aimed to analyze the open source Elasticsearch, Logstash and Kibana (ELK) stack to capture, transform, enrich, store, index, select relevant time slots and generate graphs that were integrated in a dashboard for combined visualization and analysis. Additionally, we explored its capacity to embed metadata from sensors and correct data based on a calibration certificate, also showing some relevant graphics. In this weather application, we observed that this set of computational tools are well suited to manage the daily difficulties in handling meteorological data and metadata.

Keywords

Meteorological observation system, sensors, data analysis, metrological metadata.

References

[1] Ritsche, M. Surface meteorological observation system (smos) handbook. Tech. Rep., DOE Office of Science Atmospheric Radiation Measurement (ARM) Program (United States) (2008).

[2] World Meteorological Organization. Guide to Meteorological Instruments and Methods of Observation. Part I - Measurement of Meteorological Variables. (2008). URL http://library.wmo.int/pmb ged/wmo 8 en-2012.pdf.

[3] Wright, W. Observing the Climate – Challenges for the 21st Century (2008).

[4] Santana, M.A.A. and Guimarães, P.L.O. and Almeida, E.S. and Eklin, T. The importance of metrological metadata in the environmental monitoring. In Journal of Physics: Conference Series, vol. 733, 012033 (2016). URL http://stacks.iop.org/1742-6596/733/i=1/a=012033.

[5] Prakash, T., Kakkar, M. & Patel, K. Geo-identification of web users through logs using elk stack. In 2016 6th International Conference - Cloud System and Big Data Engineering (Confluence), 606–610 (2016).

[6] Moore, J. et al. Devops for the urban iot. In Proceedings of the Second International Conference on IoT in Urban Space, Urb-IoT ’16, 78–81 (ACM, New York, NY, USA, 2016). URL http://doi.acm.org/10.1145/2962735.2962747.

[7] Barbaresi, A. Collection and indexation of tweets with a geographical focus. In Tenth International Conference on Language Resources and Evaluation (LREC 2016), 24–27 (2016).

[8] Nguyen, N. & Cuong, T. V. An efficient log management framework. VNU Journal of Science: Computer Science and Communication Engineering 32 (2016).

[9] Smiley, D. & Pugh, E. Apache Solr Enterprise Search Server (Packt Publishing, 2015), 3 edn.

[10] Akdogan, H. Elasticsearch Indexing (Packt Publishing, 2015).

[11] McCandless, M., Hatcher, E. & Gospodnetic, O. Lucene in Action, Second Edition: Covers Apache Lucene 3.0 (Manning Publications Co., Greenwich, CT, USA, 2010), 2 edn.

[12] Bagnasco, S. et al. Towards monitoring-as-a-service for scientific computing cloud applications using the elasticsearch ecosystem. In Journal of Physics: Conference Series, vol. 664, 022040 (IOP Publishing, 2015).

[13] Chhajed, S. Learning ELK Stack (Packtpub Co., 2015)

13] Chhajed, S. Learning ELK Stack (Packtpub Co., 2015)

Search










Combining wavelets and linear spectral mixture model for MODIS satellite sensor time-series analysis
doi: 10.6062/jcis.2008.01.01.0005
Freitas and Shimabukuro(Free PDF)

Riddled basins in complex physical and biological systems
doi: 10.6062/jcis.2009.01.02.0009
Viana et al.(Free PDF)

Use of ordinary Kriging algorithm and wavelet analysis to understanding the turbidity behavior in an Amazon floodplain
doi: 10.6062/jcis.2008.01.01.0006
Alcantara.(Free PDF)

A new multi-particle collision algorithm for optimization in a high performance environment
doi: 10.6062/jcis.2008.01.01.0001
Luz et al.((Free PDF)

Reviewer Guidelines
(Under Construction)
Advertisers/Sponsors
Advertises Media Information