Joseph Machado @startdataeng

Data stuff Free Data Engineering Fundamentals Course: https://t.co/Sp4qDLRwEB startdataengineering.com New york Joined April 2020

Tweets

2K
Followers

9K
Following

31
Likes

621

Joseph Machado @startdataeng

2 months ago

LLM code generation dramatically speeds up development until it doesn’t. If you’ve worked with AI-generated code, you’ve likely seen: * Bugs are buried in AI spaghetti code that takes longer to fix than it does to write it yourself. * Code that does something completely different than what you wanted This post shows you how to prevent that. You’ll learn to specify requirements, so LLMs generate exactly what you need. And avoid wading through thousands of lines of code to verify correctness. startdataengineering.com/post/using-llm… #dataengineering #llm #codegeneration

1 0 4 446 0

View Details

Joseph Machado @startdataeng

2 months ago

AI makes some reasonable guesses, but always verify the results. I’m working on an article to streamline pipeline creation. We still need humans in the loop. The image below recommends an ID-based delete-insert for a fact table, which will take forever to run if your data is large. Add these to your skills.md file. #dataengineering #llm

1 0 6 467 3

View Details

Joseph Machado @startdataeng

2 months ago

“I recommend this to everyone on linkedin asking "what to study" for data engineering. because the topics are so fundamental and work with literally every system (databricks, snowflake, redshift, bigquery, etc)” - One of the comments I got about my Design Patterns workshop Check it out here youtu.be/AXtcWgvCUtw?t=… #dataengineering #pipelinedesign

3 1 23 2K 32

View Details

Joseph Machado @startdataeng

2 months ago

@Tech_p001 Fair question. We need to review as we would human generated code.

1 0 1 26 0

View Details

Joseph Machado @startdataeng

3 months ago

Working on a project that teaches you how to build data pipelines following standard design practices and speeding up development with agent harness & skills! #dataengineering #dataproject

2 1 19 864 11

View Details

Joseph Machado @startdataeng

3 months ago

There are a lot of data projects available on the web. While these projects are great, starting from scratch to build your data project can be challenging. If you are wondering how to go from an idea to a production-ready data pipeline This post is for you! In it, we will go over how to build a data project step-by-step from scratch. By the end of this post, you will be able to quickly create data projects for any use case and see how the different parts of data systems work together. startdataengineering.com/post/de-proj-s… #dataengineering #datapipeline

1 0 12 716 12

View Details

Joseph Machado @startdataeng

3 months ago

I’ve spent way too much time thinking about my brand colors. Ultimately, I decided to go with what would make my readers' lives easier. A bright screen makes it hard to concentrate for long intervals. So I switched my website to use the Gruvbox theme. Does it help? Check it out at startdataengineering.com

0 0 4 464 4

View Details

Joseph Machado @startdataeng

3 months ago

In the past few years, data modeling has gone from “well-planned data model” to “dump everything in S3 and get some data for the end-user”. How do you ensure data correctness when data is expected immediately, and there isn't enough time to get the model right? For solutions, read this startdataengineering.com/post/deliver-d… #dataengineering #datamodel

0 1 7 399 2

View Details

Joseph Machado @startdataeng

3 months ago

startdataengineering.com/post/python-fo…

1 0 4 242 1

View Details

Joseph Machado @startdataeng

3 months ago

Updating my “Python Essentials for Data Engineers” post and came across this amazing data quality tool. pointblank from posit. Easy to set up and use, as any tool should be. No need to wrestle with 100s of configs/settings. #dataengineering #python #dataquality

3 1 9 692 4

View Details

Joseph Machado @startdataeng

3 months ago

One of the biggest causes of data completeness issues is not knowing whether your input datasets are complete. If you schedule pipelines guess when your upstream data may be ready, its hope driven. Learn how to use data asset-based triggers & start your pipelines when the inputs are ready. startdataengineering.com/post/airflow-t… #dataengineering #datapipeline #apacheairflow

0 0 5 423 6

View Details

Joseph Machado @startdataeng

3 months ago

Poll:

0 0 2 182 0

View Details

Joseph Machado @startdataeng

3 months ago

Which one should I use; Gruvbox or standard? Gruvbox is easier on the eyes But the standard theme (Bootswatch, Flatly) is familiar Let me know in the poll below 👇 #ux #readability

1 0 4 517 1

View Details

Joseph Machado @startdataeng

3 months ago

There are many data projects available online. While these projects are great, starting from scratch to build your data project can be challenging. If you are * Wondering how to go from an idea to a production-ready data pipeline * Feeling overwhelmed by how all the parts of a data system fit together * Unsure whether the pipelines you build are up to industry-standard If so, this post is for you! In it, we will go over how to build a data project from scratch, step by step. By the end of this post, you will be able to quickly create data projects for any use case and see how the different parts of data systems work together. startdataengineering.com/post/de-proj-s… #dataengineering #dataprojects

0 0 14 885 12

View Details

Joseph Machado @startdataeng

3 months ago

Using AI for development? Go read @htmx_org ‘ s post. htmx.org/essays/working… Take notes. My experience closely resembles his when building data stuff. 1. AI is good at debugging (only if we have sufficient logs & trace). 2. AI is great at creating tests (although I usually have to specify additional test cases). 3. AI will “sometimes” lead you down a complexity trap when implementing new features. When designing new features/interfaces, I always * Create my own plan with a clear objective and pseudocode * Then review it with AI and iterate. I found this to be much more time efficient and less burnout-y #dataengineering #aidevelopment

0 0 0 356 1

View Details

Joseph Machado @startdataeng

3 months ago

When choosing a tool/framework, always pick the one with the simplest implementation. But that could be extended with libraries, Plugins, Ecosystem, etc. Here are some of my favorite tools 1. DuckDB: SQL, enhanced with extensions 2. Python: English-like syntax, enhanced with libraries 3. Quarto: Markdown-like syntax, enhanced with extensions 4. nvim: vim syntax, enhanced with extensions The fancier/esoteric the tool, the harder it is to get productive quickly. What are some of your favorite tools? #tools #dataengineering

0 0 5 653 0

View Details

Joseph Machado @startdataeng

3 months ago

This is part of my free data design pattern workshop (next Sat), where I cover this with exercises & arguments. Come join me, sign up here to get notified: startdataengineering.com/newsletter.

0 0 0 212 1

View Details

Joseph Machado @startdataeng

3 months ago

90% of all facts and dimensions follow a similar design pattern, with some differences in business logic. This messy middle (fact & dims) is where DEs can lay the foundations for reliable data. If you mess this up, your team will always face issues/questions/mismatches, etc. Here is a V1 of this design

1 2 3 587 1

View Details

Joseph Machado @startdataeng

3 months ago

Whenever I design a workshop, I use @robfitz ’s “The Workshop Survival Guide”. I see how great user design increases engagement, recommendations, and leads to higher ratings. Here is a very rough version of a workshop that I am working on (coming soon). Focus on 1. Takeaways for the audience. Not just watch and forget. 2. Increase play. Interweave code exercises and visuals to help understanding. #dataengineering #userexperience