Data Collection and Storage
Xuebin Wei, Xinyue Ye · 2024
This chapter focuses on data analytics&s; first and second components: data collection and storage. The Cloud-based Data Storage section compares data storage solutions, including relational databases, data warehouses, data lakes, and NoSQL databases. The section describes the basic principles of relational database design, types of relationships, and the Entity-Relationship diagram. It then demonstrates how to set up a PostgreSQL database instance with AWS RDS. The section also introduces different types of NoSQL databases and emphasizes MongoDB data clusters&s; data structure and infrastructure. It then teaches deploying a free MongoDB cluster on MongoDB Atlas. The Data Collection via APIsection presents examples of collecting Census and Twitter (now X) data with Application Pprogramming Interfaces (APIs) and inserting the collected data into a PostgreSQL database or a MongoDB cluster. The demo codes are also published on the textbook&s;s GitHub repository.