![]() A data point (23 degree at 10:30am), or. ![]() A company can have more than one data sourcesĪn atomic unit in data flow from one data source to another data source.Has 1 or more outputs (downstream data source).Has 0 or more inputs (upstream data source).A sensor, person, application that generates data package (s).It means that data lineage will not impact the existing data flow, nor become bottleneck. Data lineage verification is an add-on on the side of the main data-flow (Technically it is possible to have permission control of the verification process, it means that data provider has to response to the ad-hoc verification requests) 3. Data provider should not be bothered by this process. It means that all verification information should be available to public. The verification process of both data integrity and data lineage should be self-service. Data integrity is the foundation of data lineageĭata Integrity is the prerequisite of Data Lineage, and they can be addressed separately. The high level description can be found at here.īesides, some deep dive information of Tangle transaction and MAM can be found at: Masked Authenticated Messaging (MAM) was introduced by IOTA in Nov 2017. This is not an article of introducing IOTA, but you can learn more from and and IOTA channel in Discord.īut most importantly, it brings Masked Authenticated Messaging (MAM) which fits into our need for data integrity and data lineage. Full scalability: Thanks to tangle data structure.This way machines can pay each other for certain services and resources. Zero Transaction Fee: Machine to Machine micropayments.This data can be made visible to certain parties. Data Integrity/Security: All data cryptographically secured in a Tangle.Therefore, IOTA becomes an outstanding DTL compares to other blockchain platform, by offering the following features: ![]() send 1000 data points from one device to another device per minute) Need to use DLT to handle large volume of transactions within a short time period (e.g.data integrity information of thousands of sensors) Need to use DLT to store large amount of data (e.g.We need a neutral and trustworthy 3rd-party for keeping both data integrity and data lineage information, as most likely the upstream (supplier) and downstream (consumer) are different organization.ĭistributed Ledger Technology (DLT) shows its potential capacity to become the neutral and trustworthy 3rd party in data lineage world, as it has the following key features:īut not all DLT are suitable for Big Data or IoT scenarios, when we have, for example, following requirements:.Data lineage is built on top of data integrity, which has to be solved first.Also, protecting logical entities is even harder in the cyberspace, both in data transportation and storage.One example is, tracking every single piece of coal from a carrier ship sounds crazy, but tracking every data signal from thousands sensors from the very same ship is quite common. The granularity of data has much more detailed scale in IoT and Big Data world.We are tracking logical entities (bits, files or data streams) instead of physical entities (coal, car parts or packages).However, compares to the traditional industries, data lineage are facing new challenges. You can easily compare this with the traditional supply chain of raw materials in manufacturing industry and/or logistic industry. It also helps to trace errors back to the root cause, and comply with laws and regulations. It increases trust and acceptance of result of data process. The goal of data lineage is to track data over its entire lifecycle, to gain a better understanding of what happens to data as it moves through the course of its life. ![]() Various data sets are generated (most likely by sensors), transferred, processed, aggregated and flowed from upstream to downstream. If we say “Data is the new oil”, then data lineage is an issue that we must to solve. For example, Timestamp is now an optional field.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |