Replacing Myself with AI
Could it really be this simple?
Last year I wrote a post detailing that my Audiobook pipeline wasn't really *passive* income, going so far as to say I thought that [passive income is a scam](https://www.joshbhattarai.com/passive-income-is-a-scam/). My project was making $300 a month, but it took 10-20 minutes of manual work every day, never mind the whole year working full-time on development. I had felt that the juice wasn't really worth the squeeze to continue, and I wanted to be done.
The Task
Humans are effortlessly intelligent
The task was simple enough. Every day, I find a Project Gutenberg and LibriVox pair of a book. I ensure the edition/translation/volumes match, and then I download the audio files and the text to a directory. Then, I need to ensure that the single contiguous text of the ebook gets properly split at the same spot that the human narrator on Librivox decides to start and stop each mp3 file of the audiobook. It seems simple enough but is quite a tedious and repetitive task. Librivox recordings always start and end with a disclaimer., scrubbing to the correct moment on the audio file taking care to listen to the first/last couple words of the narration and NOT the disclaimer is quite difficult when over/undershooting by a few seconds results in treading into the disclaimer audio. Then I try to find those few matching words and hope they're unique to the one spot within the ebook., Otherwise, I risk splitting on an unrelated section of the text and bungling the video for all future readers. And I need to run this, every. single. day.
The thing is, this task is complicated even when I try to describe it over text. In reality, many complex tasks are being handled behind the scenes. I am listening to audio recorded by a random volunteer, often using lower-quality equipment, and mentally transcribing it into text. At the same time, I have to distinguish the actual audiobook narration from the LibriVox disclaimer and consciously ignore the disclaimer each time.
Next, I search the ebook text for the corresponding passage, keeping in mind that the transcription may contain errors, including misspelled words, mispronunciations, or sections that were skipped entirely during the narration. Once I locate the correct passage, I insert the delimiter and move on to the next section.
The process turned out to be far more complicated than I had originally thought.
1. Find a Relevant Title
2. Match a text and audio source
3. Download both
4. Listen to the first chapter's MP3 file.
5. Determine where the disclaimer ends
6. Identify first words of non-disclaimer audio in the MP3 file
7. Identify the first words of non-disclaimer audio in the ebook text,
9. Identify and ignore transcription errors
10. Find last words of non-disclaimer audio in the MP3 file
11. Find the last words of non-disclaimer audio in the ebook text, ignoring transcription errors
12. Input the delimiter into the text
13. Rinse and repeat.
Luckily the AI revolution just happened. I spent some time deliberating if it was worth it, but doing the manual work every day was starting to get too annoying. I thought of the opportunity cost, everything that I could be doing with the time and mental fatigue that wen into the every day manual work of Ion books. I decided to pull the trigger.
Translating the Task to Compute
The puzzle behind software engineering
First I had to use YouTube's API to figure out which titles I had already posted to Ion Books. No point in wasting compute to upload the same title twice. Luckily I had been creating playlists for each title since I first started Ion Books, a forward-thinking decision I made 5 years ago that would pay dividends today. By retrieving the playlist title of every playlist on Ion Books and doing some regex, I had a list of every title I had uploaded. After that, it was a simple system prompt that would instruct any major LLM to successfully suggest a new title to upload. Then, a simple Claude skill to match the Librivox page to a Gutenberg ebook page. Then a simple set of Python scripts to download the content from both pages to a directory.
The next step was listening to the MP3 files. Listening to an MP3 file is not something computers can do. All the computer sees is two waveforms; how does it know which ones corresponds to "This is a Librivox recording..." versus "All happy families are alike..."? The answer is WhisperX in my case. The first task was taking each of the MP3 files and generating transcripts using OpenAI's open-source transcription model WhisperX.
WhisperX was able to transcribe the MP3 files at ~4x real-time factor. I was in good shape! I had designed a pipeline that automated steps 1-6 of the original 12-step problem. However, upcoming was the most challenging technical piece: aligning the transcripts to the ebook text, all while avoiding disclaimers, transcription errors, shoddy microphones, and a litany of narration mistakes and accents from the good volunteers of Librivox.
"Hiring" Top Talent
Computers have come a long way
Once I had my transcripts in hand, I decided to do a proof of concept. Pass the full plaintext of "Thus Spake Zarathustra" and the full transcription of the entire 16-hour corresponding audiobook into Gemini 3. To my great surprise, it gave me perfect output. I was over the moon! I had automated a task that ate into 10-20 minutes of my time every single day of the last 5 years of my life. Then I saw the cost. This one job had cost $4.33 worth of Gemini tokens. To this day, my YouTube video for Thus Spake Zarathustra has made $4.24 off of AdSense, and is one of my more evergreen titles. It was easy to see that this was not going to be profitable. Just automating this task was not enough; I now needed to automate efficiently.
Budget Cuts
Even LLMs have to be used strategically
The easiest optimization available was a sliding window. I knew with 100% certainty that if I combined all of the transcripts I would have a match with my ebook, disregarding transcription errors and Librivox disclaimers. Matching the first MP3 file's transcript to the ebook's plaintext would allow me to then remove the first chapter's plaintext from all future queries, reducing token costs. This turned out to be EXTREMELY difficult. One early mistake of the LLM would cascade catastrophically down through the book. A dedication from the author, or a skipped table of contents, would cripple the alignment's fragile heuristics and leave the final alignment a garbled mess. I needed something better.
I turned back to AI: I had Claude launch Deep Research into similar alignment problems. Claude helped me to find the Carnegie Mellon University Sphinx project, using MFCC, hidden Markov models, and other machine learning strategies to perform forced alignment before LLMs. Using CMU Sphinx's strategies, I was able to reach ~80% accuracy on my alignment pipeline before an LLM ever got involved. This drastically increased speed, reduced cost, and I was able to Gemma 2 on the Gemini free tier to reach 90% accuracy. A huge win for my pipeline.
Still, the alignment was not good enough. The alignment was integral to production. The front matter and Librivox disclaimer stripping part of the log resulted in the very first thing an end user saw after clicking on one of my thumbnails. If the user saw a cut-off sentence or half of a table of contents, the whole video would reek of low energy, and my channel would suffer. I needed higher accuracy.
I tried many different AI models, and the only model that reached my cost to accuracy ratio was the 32 billion parameter model of Qwen3. I was astounded by how much better this Chinese open-weight model was than the corresponding open-source models coming out of American labs. It was a night-and-day difference. My 90% accuracy skyrocketed to 97%, and the cost went up to around $10 per month. I ran my inference through Groq, which provided me lightning-speed inference. I eventually even ran my transcription through Groq since the speed increase ended up saving me money on GCP compute.
Even still I noticed every couple of videos was horribly mangled. Not only were there sentences cut off on the first page of text, but the alignment drift slowly cascaded until usually the halfway point of the video was completely out of sync. I delved into the code and discovered that Qwen was struggling to reconcile the table of contents, author dedications, footnotes, prologues, epilogues, etc. I tried everything: system prompt tweaks, widening the window of the ebook to provide to Qwen at the cost of increased tokens, custom system prompts for the first and last chapters of the books. Nothing worked.
Finally, I decided to consult the documentation of the Unsloth project and explore fine-tuning. This was quite the big undertaking. Unsloth suggests that for a 32B model to have at least 50,000 training examples, (trending closer to 100,000), to have good results. Luckily, over the last 5 years I've kept a meticulous archive of the source files for nearly every book I've uploaded. Five years of uploading books every single day, (365 * 5 = 1825 books--if we assume 20 chapters per book), means 36,500 gold-standard human-grade examples of chapters aligned. Still only a bit past half the "bare minimum".
This was a textbook use-case for synthetic data generation. Using my ground-truth training examples, I could turn "All happy families are alike" into "happy families are all like" to simulate narration/transcription errors and turn my ~35,000 odd training samples into 300,000 training samples. The process of turning my dataset into a properly formatted AI training set took about 8 months. I bought a new $2000 PC to support a new GPU to run the training pipeline faster. I tore my hair out trying to balance different versions of WhisperX, Unsloth, transformers, and Qwen inference in my Dockerfile. Then, I finally had my model file, and when I evaluated it on my training data... nothing. From base Qwen3's 97% accuracy, I got to 97.5% accuracy.
AI Outsourcing
Rediscovering outsourcing
I had spent a little under a year trying to save myself time. This was like my experience with trying to generate passive income: something that was supposed to save me time, outsourcing my workloads to AI, was taking me more time than it was worth. I paid for the Claude monthly plan. I had a headless Claude code agent audit the work Qwen3 did. I had Claude take over the most difficult tasks, but delegated the rest to Qwen. The accuracy jumped up to 99.4%. It was finally in a place to release.
Currently, I pay $15 a month to Groq for transcription and Qwen3 alignment, $100 a month to Google for the GCP video render compute, and $100 a month to Claude to handle the trickier parts of the alignment. Claude by far handles the smallest portion of the work, but it is also the most complicated. It also takes a "management" role over the Qwen jobs. I find it ironic that I have somehow invented overseas outsourcing again. Where in my own day job I serve as the architect planning features and overseeing a team of cheaper engineers in India, my AI pipeline has a significantly more expensive American AI overseeing multiple cheaper Chinese AI jobs.
There may still be a way to set up Qwen juuuust right so that I don't need Claude.The revenue increase from Claude is worth the $100 a month price tag though. Ion Books now makes ~$800 a month, up $300 from this time a year ago. Paying for Claude, Groq, and GCP to take myself out of the equation permanently is worth it.
Growing Pains
Passive never means passive
These days, I get an email at 6:30PM. "On the Eve" by Ivan Turnegev will be published tomorrow, here is the thumbnail generation prompt. My work for the day is copying the thumbnail generation prompt, pasting it in Ideogram, and then choosing one of the four generated images as the thumbnail. I've cut my work down from 10-20 minutes of fully immersed focused work into a 1-2 minute task.
I say it's almost completely automated, but that's only when the pipeline works. I've had to deal with Groq deprecating Qwen2.5, GCP deprecating gsutil, tokens expiring, silent failures that I thought would get logged to me in an email, but in actuality halted the pipeline for two weeks. I've switched daily routine, episodic manual work with sporadic technical debugging work. But if I draw out the parallels to outsourcing, isn't this a good thing? Even though I replaced myself with AI, I'm still finding my technical skills useful, something the rest of the software industry seems to be figuring out.