Mathematics of Data Engineering

My love for math has transferred to data engineering

Hello everyone, I have been having some days where I have been thinking of how my bachelors degree is applicable to my current career path.

When I was a math teacher, it was pretty straight forward. I was just teaching basics of algebra to the functions in algebra.

With my background, I could tell the deep knowledge of mathematics was very helpful. I could hold deep conservation with students about the subject that loved so much.

Once I moved to data analytics, it wasn’t so straight forward. I don’t count statistics as mathematics. Statistics is a sister to math but isn’t it. As a data analyst, that’s what I was doing average, median, and counts.

I started to build tables with SQL, this is where my love for mathematics came back. The mathematics that was mostly used was linear algebra.

Linear Algebra Used

---

Linear algebra is where they introduced matrices and vectors. For example, I was working in an interesting problem lately where I need to transpose a table so there was a single row for each person ID within the table.

Let’s go over some mathematics first before explains how I did it in SQL. Let’s start off with a 2D matrix with any width or length.

my caption

As you can see the object in the top left hand corner is a(1,1) and the one next to it is a(1,2). Let’s abstract this even more to just a(x,y) for any object in a matrix.

In a transpose, you just flip the x and y. When you do that, the object changes position.

Let’s say there was a(1,2) = 2 and a(2,1) = 5. Then we transpose. a(1,2) = 5 and a(2,1) = 2.

If you really want to understand this, please take out a pierce of paper and make a 4 by 4 matrix and put any number in each 4 spots. Make sure the number are different, then transpose it. You will see what happens.

SQL Problem at work

---

Let’s say I make a table with one primary key column and the other column has a term and course being taken. The issue here is that I don’t have a second number to make my indices from the example above. This is where I had to use a window function.

The window function I used was ROW_NUMBER.

ROW_NUMBER() OVER (
            PARTITION BY personID, term
            ORDER BY courseName, sectionID
        ) AS ClassSequence

In this function, I would get it to count every course that is in each term. NOW I CAN USE A TRANPOSE! The was I did it was with a MAX() function where I use a case when term = and ClassSequence =. Then I could position in a want to make a left join to the personID. This let me have one row per person.

Love for data engineering

---

After this problem, I just fell in love with data engineering. I got to use my abstract thinking from mathematics that I loved doing to my work again. And the mindset of a physicist where you ask what type of math can I use to solve this problem. God, that was an awesome problem to solve.

What problems have you been working on?

Thank you so much for reading my journey in data engineering and what I have learned on the way. If you enjoyed this, please think of subscribing to my newsletter.