Importance of Data Engineering
You know when ML was big and data scientists became data engineers?
Hello everyone, it’s Brien. Just your local teacher that turned data analyst. Now that I am an Engineer. I can see why my company or a lot of companies are hiring data engineers.
If you are new here, I write about data related content. This content comes from a lot of my own personal experience and what I am currently working on or realizations from my own eyes.
Data Analyst
I think this story starts when I picked up a role as a data analyst. I was given a role that applied for within the same place I was a teacher. I asked what they wanted me to do. Things came up with random questions. Such as geolocation or correlation analysis, I started to notice one thing every time.
The most time consuming part of my role was the pulling of data from our local database or API’s from software that we use in the classroom.
I thought there has to be a quicker way. This is where data engineering comes in. Data engineering is the practice of build tables within in a database to build them in a way that analyst, scientist or stakeholder can use. Not like software engineers where they store data to keep it somewhere for users to access. Data engineers have to set it up in a ways where it came be translated into charts or even into Machine Learning models.
Data Engineer
I started to build out more tables and architecture so it was in a way for me to pull it more usable. This is where I got my realization, because I spent so much more time setting up these tables. The data cleaning or transformation as an Analyst became so much easier.
This couldn’t be just used for an Analyst. This is where setting things up correctly for an AI because huge. I read somewhere that data engineers have gone up like 40%. I can understand. The value of a data engineer is time. Time saved pulling data out and cleaning it. And have less errors in predictive models or less errors in an AI agent. This all comes down to data engineering.
This is my realization for this newsletter post. Hope you are all well. Keep up the hard work.
Thank you for coming back every week from a one person data team.
Please give this like or even a share. It would really help me.