Journey from Data Analyst to Data Engineer
This is going to be about my career growth from data analyst to data engineer
Hello everyone, if you haven’t heard by the amount of times that I have posted about this, I have been officially moved from data analyst to data engineer. This is going to be my deep dive on how I did it.
The first thing is first, thank you so much for coming back every week to listen to what I have to write. Writing isn’t something that I naturally am into, but at the same time I think I have a better way of organizing my thoughts through writing. So, thank you so much.
If you haven’t read my article on how I became a data analyst after being a teacher for 5 years, you are welcome to read it from the link below:
5 lessons learned
Ever wake up in the morning knowing you were made for something bigger or at least something that aligns with yourself?
Beginning of Data Engineer work
There is a lot of talk out there about how you should only do work that you are currently getting paid for. This is where I will have to completely disagree. I think if you are interested in something and your work will allow you to do that work in your current role, you should go for it!!
Just like when I was a teacher, I started doing data analyst work for the charter school that I was working for. The same thing happened to me again when I was a data analyst. I started to get into more data engineer work. It all started from here.
WHAT we can use SQL in here
For my second year as a data analyst, I got used to how the data was being pulled from Infinite Campus. During that time, we were using their built‑in ad hoc system. There was an issue that came up where we needed data from two tables that could not join within their ad hoc system. I was looking through the place where you could schedule data to be extracted. I noticed that there was a turned‑off extraction, and I opened it. I found that there was SQL code.
I asked my boss about the SQL code. She told me it was code that she copied and pasted a couple of years ago when they needed to connect it to a system because of the Nevada Department of Education. Then I told her that we could probably have access to all the tables using SQL.
I remembered that we had that issue of not being able to join two tables. I told her that with SQL we could do it. She gave me the go‑ahead to try it out. I GOT IT TO WORK. Then after that, it was a floodgate of building lots of SQL code to EXTRACT and TRANSFORM the data. From building SQL for our medium‑size charter school, I changed having 5 data extractions into 1 beautiful SQL pull.
The joy of API’s
I was teaching myself about APIs and how they work. I was learning that I could pull data from them. During that time, and even now, I was pulling data from Canvas. We became interested in submission data and how students were doing. We found out that students were only doing work at school even though they had access at home.
Then my boss at the time — and still my boss now — asked me if I could set up a schedule to pull that data. So I started to deep dive into software engineering practices. At that point, I had deployed a Cloud Run script to help make Smartsheets better. Thank the lord we got away from that software. Anyways, I set up code in Cloud Run on Google Cloud to pull the data from Canvas’s API. I had to use Cloud Scheduler, where this took a deep dive into understanding CRON.
I had to reach out to my family friend who was a Director of Software Engineering. He sent me over a document that he put together to teach myself Python programming. This is where I started learning about software engineering and best practices with environment variables and unit testing.
Changing Data Platforms
At this point in my journey, I was still a Data Analyst with many data pipelines built from Python and SQL. To add more fire to it all, my boss wanted to change data platforms from Smartsheets to Airtable. I don’t think I’m in a place where I can explain why, but one of the many reasons was because we were hitting data capacity issues. Smartsheets was starting to become too small for our organization, and it was slow as well. This is where my boss told me that I would be the head person for this project of switching platforms. Data Migration. She did help as well, but it was a complete solo mission.
I was messing around with Airtable to see if it was a good data solution for our place. Then we went all in, and I had to learn how to connect data from Infinite Campus to Airtable. Originally, we would push it into an FTPS server and then reflect it into Google Drive. But… Airtable doesn’t ingest data from Google Drive. It’s made for holding data in it, and then you push it out.
I had to go back to Infinite Campus and see if they could extract data besides FTPS. WAIT A MINUTE. I saw that it said HTTP POST. I have seen this before!!
With the new knowledge of teaching myself about APIs, I had to build my own RESTful API. I taught myself a framework called FASTAPI because my family friend said there are plenty of people out there who say it’s fantastic. I went to YouTube and built a simple FastAPI app.
I had to learn how to let this app have a POST route to authenticate the person posting to this app — Infinite Campus. I needed another route to take in a CSV file. So I set up these routes on my app. But for some reason, it was taking a while. Then I learned that you could run functions in the background while our POST would hit the endpoint.
The other thing I had to teach myself was a load balancer with a firewall. This is where you don’t want to have an exposed app, because this can drive up costs for you on the cloud.
Title Change
I finally went to my boss and told her the plan for this new integration from system. The data transformation will stay on Infinite Campus and I built a backend app to take the data and move it into Airtable.
This is where we both started to agree that most of my work was engineering now and not so much analyst work. She wanted the Title to be Solutions Engineering because I provide technology solutions to the business. I just asked her if we could put Data in front of it. Because I know in the market place Data Solutions Engineer is closer to what I was actually doing.
You know just in case I plan on leaving or get recuited somewhere else.
Thank you so much for making it to the end. If you enjoyed reading my journey from data analyst to data engineer. You are welcome to subscribe to get more updates about things that I am learning or updates about my career.