After over a decade of working as a backend engineer, I switched to a new domain. This post explores my unexpected transition into the world of Big Data and the motivations behind my career shift.
My background
I’m a software engineer who has worked in several companies for 13 years. For most of my professional career, I mainly stayed inside the good old JVM (Java, Kotlin, Spring Boot, Swing, JavaFX, etc.) and rarely worked with some other technologies (C, C++, Flutter). Especially for the last six years, I worked on a web project using Java and Spring Boot, deployed on the AWS, and kept it up 7/24, taking on-call responsibilities as needed.
My current job
At the beginning of this year, I changed my career and started working in the big data domain using the Scala language. My responsibilities involve developing the open-source Apache Spark data processing engine: SQL plan optimizations, cluster manager implementations, Kubernetes integration enhancements, etc.
This is unexpected!
This was a very unexpected move for me, as I could understand looking to my friends’ faces when I explained the decision 🙂 It did not seem to be the safest and most comfortable path to take because I would need to learn a completely new domain, a new language paradigm, and a new project that is fully packed with many engineering thoughts and optimizations.
Understanding my push and pull forces
Why I made this “radical” switch?
First of all, to make a career move, I need either a push or pull force or a combination of both. Typically, I need to feel increasingly uncomfortable in my current job before I take action. Otherwise, I tolerate minor inconveniences and will preserve my current position until it exceeds my inconvenience threshold. This inconvenience is the “push force” for me.
As you would guess, the pull force is an attractive opportunity. I need a strong combination of both: a strong push, a strong pull, or a strong sum of push and pull forces.
My push force
For a while, I felt I was merely tolerating inconveniences in my previous job. Because I did not feel like learning new things and solving engineering problems. A big part of my job became customer support. Besides, I had to take on some dull responsibilities, such as tracking country regulation changes and collecting necessary documents inside the company to make sure our notification services continued to work without any legal or technical issues. I was not feeling like an engineer anymore, negatively impacting my motivation.
But I didn’t understand this problem and kept going without energy. I tended to preserve my job because it paid relatively well, and I feared a possible change. I preferred to stay inside my comfort zone. Towards the end of last year, the job and pay became unattractive.
This growing dissatisfaction led me to explore new opportunities.
My pull force
On the “pull-force” side, I believe there are two important topics to work on to become future-proof nowadays: AI and security. AI will bring endless possibilities and change the world dramatically in the upcoming years. It will happen gradually but faster than we expect. On the other hand, security is always an important and attractive topic. With the rise of AI, security will become much more important because AI will bring more and bigger security challenges in addition to its benefits.
So, I was (and still am) excited by the rise of AI, its incredible uses, and its endless possibilities. I followed multiple AI newsletters to stay up-to-date about what was happening worldwide. I constantly found myself watching some AI-related videos on YouTube. Yes, this was the pull force for me.
All you need is a stronger push.
With a stronger push force and a slightly stronger pull force, I finally had the courage to go where I wanted to be. Sometimes, all we need is a stronger push force. I wish I had gotten that earlier so I didn’t waste time mindlessly.
Quit timely!
Anyway, I started applying for job postings. It is funny that I was still applying to similar roles even though I already hated them. I was afraid to go down a different path due to the lack of confidence I inherited from being unmotivated, low-performing, and wasted time. Friendly advice: Don’t be late to quit when you see the signs.
Imposter syndrome
My ultimate goal was to work on one of the futuristic topics, AI or security, but due to my lack of confidence, I thought I couldn’t switch to an AI job directly, especially when it is in demand. No one would give me such a job. Maybe I was wrong. But I thought the next best thing would be to work in the big data domain first, which is a domain close to AI, and I hope to jump on the AI wagon someday in the near future.
Among the other jobs, I also applied for the job I have now, which is a big data job, per my understanding from the job posting. I had a few long technical interviews with several experienced engineers. I enjoyed the process. Luckily, they don’t look for direct knowledge of the technology (e.g., Apache Spark, Scala), which I had no idea about. Still, they looked for the algorithmic fundamentals needed to perform the job. I’m happy they did so; the interviews went well for me. Finally, I got the job.
Coming back to life
After feeling unmotivated, I opened a new page in my file. There was no hook to the past, no dull responsibilities; it was exciting technology, and I was starting from scratch! It was relieving.
Of course, that wasn’t a one-day switch. I had inherited some negative thoughts about myself. It takes some time to gain self-confidence. Also, everything was new and unknown to me.
But eventually, I became more familiar with the domain and appreciated my decision. Every day, I learned new things that were both exciting and valuable to my career.
Now, I’m happy with the job, which offers endless learning opportunities and allows me to apply my design and engineering capabilities. Apache Spark consists of well-engineered parts. You can learn many things just by reading the code. You are what you consume, so reading good code will make you a better developer. Scala’s functional programming paradigm has also opened my eyes to new ways of thinking.
Lessons learned
I was late to quit, even though I felt it inside. This wasn’t the first time I quit later than the optimal time. Note to myself: Don’t be late next time!
Don’t ignore your instincts. Recognize when you are feeling unfulfilled and take action to fix it.
Sometimes, discomfort could lead to growth and better opportunities. Don’t fear the change; follow your passion even if it seems risky.
Have you ever faced a similar crossroads in your career? What influenced your decision-making process? I would love to hear your stories in the comments below!
Thank you for reading!