Every minute a developer spends waiting is a minute of lost innovation.
Recently at Taboola, a critical question has emerged: how can we tackle the productivity challenges and inefficiencies that weigh down our developers?
The first step was to identify the key areas needing improvement. In a monorepo setup such as the one we use at Taboola – with over 8M lines of codes and 160K unit tests – these challenges often center around long feedback loops and delayed test cycles.
While monorepos provide consistency and foster cross-team collaboration, they also introduce latencies and lengthy pipelines that can slow down development. In this article, we address some of these bottlenecks and the solutions we’ve implemented to reduce delays and enhance developer productivity.
Current Development and Deployment Workflow
At Taboola, when developers push their code to a Git branch, an automated pipeline kicks off a build process that takes an average of 15 minutes. This includes compiling source code, generating JAR files, building Docker images, and packaging artifacts, as well as running unit tests to catch issues early.
After the build completes, the newly created Docker images can be deployed to testing environments, where developers can execute integration tests to validate functionality and ensure stability before progressing to further stages.
This process repeats iteratively until the changes meet functional and code quality standards. If tests fail, developers refine their code, push updates, and trigger another build. These cycles of testing, reviews, and fixes can be time-consuming, highlighting the need for optimization before the code is verified and ready for production.
Remote Webpack: Reducing Feedback Loops for Frontend Development
The Problem
For a frontend developer, the feature development process follows a similar pattern: opening a branch, adding the code, and triggering a Jenkins build. Once artifacts are ready, the next step is the creation of an on-demand testing environment in Kubernetes with the newly built version, where Selenium tests are executed to simulate user interactions with the application, ensuring that the frontend behaves as expected across different browsers and environments. This feedback cycle introduces significant latency, as the considerable wait time between code changes and testing can disrupt the development flow.
The Solution
When faced with these productivity challenges, it’s tempting to seek new tools or technologies. However, some of the most impactful solutions can come from optimizing existing tools.
In our case, IntelliJ Gateway was the starting point. This tool allows developers to access fully configured IDEs in remote environments, reducing startup and indexing time while preserving IntelliJ’s powerful features. We leveraged IntelliJ Gateway’s remote development feature to connect directly to a development pod running in Kubernetes. This pod is provisioned with a full-fledged development environment and the connection is established over SSH, with IntelliJ Gateway handling the remote execution of the IDE backend, while rendering the UI locally.
We extended this approach with rsync, synchronizing changes between a developer’s local machine and a remote pod in the same namespace as the Webpack service. This remote pod processes changes immediately after each recompile as it efficiently transfers only the modified files to the remote pod, creating an experience that feels just like working locally.
To streamline access, the remote pod is externally monitored, ensuring that one is always available when it’s needed. Developers can instantly resume their sessions without needing to build again in Jenkins or create a new pod. This setup provides a near-local development experience while benefiting from the scalability and consistency of a Kubernetes-hosted environment.
Key Benefits
- Immediate Feedback: Developers get near-instant feedback after each code change, eliminating the overhead of full rebuilds.
- Improved Resource Utilization: Resolves network issues while reducing CPU and memory usage on local machines.
- Streamlined Setup: While the initial setup takes 8–10 minutes, each subsequent change provides immediate results, keeping the developer workflow smooth and uninterrupted.
| Step | Traditional Workflow | Optimized Workflow |
| Code Change | Edit file locally | Edit file locally |
| Build Time | 15 mins in Jenkins | A few seconds |
| Deploy | 7 mins on average | No additional deployment |
| File Sync | Full artifact upload | Only changed files |
| Test Feedback | Delayed (after deployment) | Immediate |
This feature transformed the feedback loop into a seamless process, allowing developers to focus on what truly matters: writing great code.
Gonzales: Testing in a Production-Like Environment
The Problem
In traditional workflows, developers often rely on staging or test environments that may not fully replicate the complexity or scale of the production system. As a result, features might behave differently once deployed to production, leading to bugs or unexpected behaviors. Additionally, testing in these environments can be slow, requiring significant setup time and resources.
The Solution
Gonzales allows developers to test feature branch changes in a production-like environment within just a few minutes. The way it works is simple yet powerful: Gonzales copies the local changes from a developer’s feature branch into a production machine, where the application is then run in read-only mode. This mode, a feature of the underlying system, ensures no side-effects like traffic logs or data manipulation occur while still running the application in a production-like environment.
This approach allows developers to manually test their changes without affecting live user data or generating pageviews. The code under test is packaged in a JAR used exclusively when running the app with Gonzales, ensuring that it doesn’t leak into the production system, even though it operates on the same machines.
Key Benefits
- Mirrored Traffic: By exposing the test machine to the same incoming traffic as the production environment, Gonzales provides an accurate reflection of real-world behavior.
- Safety and Isolation: Since the environment operates in read-only mode, no side effects (such as data manipulation) can occur, ensuring safe testing.
- Efficiency: This approach dramatically reduces setup time, while offering a reliable testing environment, bridging the gap between staging and production.
By simulating production conditions with real traffic, Gonzales ensures that features are thoroughly vetted before deployment, reducing bugs and enhancing confidence in production releases. By optimizing our development and testing workflows, we’ve removed key bottlenecks that were slowing down feature delivery.
Remote Webpack has accelerated frontend development by providing immediate feedback, while Gonzales ensures reliable testing in a production-like environment without the usual setup overhead.
As organizations strive to enhance developer productivity, evaluating and refining existing workflows can be a highly effective approach. We encourage teams to audit their development processes, identify bottlenecks, and explore optimizations that maximize efficiency without adding unnecessary complexity.
Credits: Alon Pilberg and Shirel Hadad
