Is Your Mobile Number Being Illegally Traded Right Now?

[Mostafa044]

Purpose: To capture data from various sources and move it into the data lake. For "recent" mobile data, real-time streaming is crucial.
Batch Ingestion: For historical data dumps, daily logs, or less time-sensitive data.
Tools: Apache NiFi (open-source, visual data flow management, good for integrating diverse sources), Apache Sqoop (for RDBMS data), custom scripts.
Streaming Ingestion: For real-time mobile events (CDRs, app clicks, MFS transactions).
Tools: Apache Kafka is the industry standard for high-throughput, low-latency streaming. It can act as a central nervous system, decoupling data producers from consumers. Other options include Apache Pulsar.
Considerations for Bangladesh: Given potentially less stable internet infrastructure in some regions, Kafka's fault tolerance and ability to handle bursty data are beneficial. Open-source solutions reduce licensing costs.
Raw/Landing Zone (Bronze Layer):

Purpose: Stores data exactly as it was ingested from the source, malta phone number list without any transformations. This serves as an immutable, historical record and a single source of truth.
Format: Native formats (JSON, CSV, Avro, Parquet, log files).
Storage:
On-Premise (Cost-Effective for control): Apache HDFS (Hadoop Distributed File System) or MinIO (open-source S3-compatible object storage, great for self-hosting). This offers more control over data locality, potentially good for compliance and reducing cloud data transfer costs.
Cloud (Scalability & Managed Services): If a cloud provider is used (AWS S3, Azure Data Lake Storage Gen2, Google Cloud Storage), these are highly scalable and managed solutions. However, consider the cost implications for data residency and transfer.
Considerations for Bangladesh: For initial deployments, self-hosting with HDFS or MinIO can be highly cost-effective, leveraging commodity hardware. However, it requires significant operational expertise.
Refined/Staging Zone (Silver Layer):