LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
data-extraction document-processing etl-pipelines unstructured-data-extraction api-deployments open-source-data-pipeline
- Updated
- Python
#
etl-pipelines
Here are 44 public repositories matching this topic...
Fast and efficient unstructured data extraction. Written in Rust with bindings for many languages.
nlp rust pdf machine-learning natural-language-processing ocr etl tika extraction docx data-pipelines pdf-parser unstructured unstructured-data rag etl-pipelines llm
- Updated
- Rust
The simplest way to scale Python.
- Updated
- Python
Data pipelines from re-usable components
data-science sql etl pipelines immutability data-engineering functional-reactive-programming data-analysis data-pipelines data-pipeline etl-framework etl-pipeline etl-pipelines
- Updated
- Python
The open-source Useful SDK. One python decorator in the Useful library allows for full observability of Python functions within an ETL.
- Updated
- Python
Data Cleaning for Pyspark
- Updated
- Python
A project structure for doing and sharing data engineer work.
scraper etl cookiecutter python3 data-engineering data-extraction cookiecutter-template etl-pipeline etl-pipe etl-pipelines
- Updated
- Python
Lien de l'application
python docker dockerfile airflow mongodb etl docker-compose makefile postgresql orchestration data-engineering atlas extract-transform-load mongodb-atlas etl-pipeline streamlit etl-pipelines streamlit-dashboard duckdb
- Updated
- Python
DataSift auto applies a data pre-processing pipeline to Data Science Projects.
- Updated
- Python
End-to-end Azure Data Factory project transforming raw sales data into customer-level insights using pivot transformation and storing results in Blob Storage.
data-transformation data-engineering blob-storage azure-data-factory etl-pipelines mapping-data-flows pivot-transformation
- Updated
This project demonstrates a comprehensive data warehousing and analytics solution, from building a data warehouse to generating actionable insights. Designed as a portfolio project, it highlights industry best practices in data engineering and analytics.
- Updated
- TSQL
Big Data ETL pipeline for Brazilian e-commerce data. Implements data ingestion, transformation, and storage using Apache Spark, Hadoop, and SQL. Designed for scalable data processing and analytics.
- Updated
- HTML
Build ETL piplines on AirFlow to load data from BigQuery and store it in MySQL
- Updated
- Python
Complete portfolio of data engineering projects from Udacity's Data Engineering with AWS Nanodegree.
apache-spark relational-databases nosql-database airflow-dags etl-pipelines lakehouse-architectures automated-workflows-using-aws-services
- Updated
- Jupyter Notebook
Modern Data Warehouse and Analytics Project implementing Medallion Architecture (Bronze, Silver, Gold) with ETL pipelines, SQL data modeling, and analytical reporting.
data-science sql sql-server etl data-warehouse sql-query data-engineering data-analytics datawarehousing data-analysis sqlserver data-science-portfolio datawarehouse data-cleaning datalake data-warehousing etl-job etl-pipelines data-lakehouse medallion-architecture
- Updated
- TSQL
🗄️ IBM Relational Database Administrator with GenAI Certificate Portfolio – A comprehensive collection of projects, labs, and assignments showcasing expertise in relational database administration, 🏘️data warehousing, 🔁ETL pipelines, and 🤖Generative AI integration for modern database management.
mysql linux sql postgresql data-engineering database-administration erd relational-databases apache-kafka data-modeling shell-scripting data-warehousing apache-airflow star-schema backup-restore snowflake-schema database-security etl-pipelines generative-ai sql-optimization
- Updated
- PLpgSQL
🚀 A comprehensive showcase of projects and skills from the IBM Data Engineering Professional Certificate! 📚 Features include: 🔄 ETL pipelines, 🗄️ data warehousing, ⚡ big data processing with Spark/Hadoop, 🛠️ database administration, and 📈 business intelligence dashboards. Built with 🦾 to demonstrate real-world data engineering capabilities!
mysql python machine-learning sql kafka big-data apache-spark mongodb hadoop nosql postgresql databases data-visualization data-warehouse data-engineering business-intelligence data-analysis data-pipeline apache-airflow etl-pipelines
- Updated
- PLpgSQL
Master the AWS Data Stack! 🚀 This repository features 15+ Industrial Data Engineering Projects covering Serverless ETL, Real-Time Streaming, & Data Warehousing. Hands-on labs for S3, Lambda, Spark, Airflow, Snowflake, Redshift, Kinesis, & Glue. Includes production-grade CICD pipelines. A complete roadmap to becoming a top Data Professional.
aws aws-lambda snowflake pyspark data-engineering amazon-kinesis amazon-dynamodb cicd amazon-redshift real-time-streaming amazon-s3 amazon-athena apache-airflow aws-step-functions aws-glue github-actions delta-lake etl-pipelines
- Updated
This repository contains my first end-to-end Data Engineering project, built using Microsoft Azure Cloud and Azure Databricks with PySpark.
data cloud spark azure pyspark data-structures data-engineering databricks microsoft-azure delta-lake etl-pipelines lakehouse data-lakehouse medallion-architecture lakehouse-architectures
- Updated
- Jupyter Notebook
Improve this page
Add a description, image, and links to the etl-pipelines topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the etl-pipelines topic, visit your repo's landing page and select "manage topics."