:zap: A distributed crawler for weibo, building with celery and requests.

基于 python 3.9.13 + django 2.2.16 + channels 2.4.0 + celery 5.2.7 + ansible 2.9.14 + paramiko 2.6.0 + AdminLTE-3.0.0 实现的运维 devops 管理系统。

This project provides a comprehensive data pipeline solution to extract, transform, and load (ETL) Reddit data into a Redshift data warehouse. The pipeline leverages a combination of tools and services including Apache Airflow, Celery, PostgreSQL, Amazon S3, AWS Glue, Amazon Athena, and Amazon Redshift.

Establish architecture-as-code practice using C4 modeling and Structurizr DSL. Use after bootstrap-project, when establishing architecture documentation, or when migrating from diagram-based to code-based architecture.