Understanding the Trend of In-House Development
Context
In the tech industry, the question of whether to use off-the-shelf solutions or create custom ones often arises. Big tech companies like Yandex, Google, and Facebook frequently choose to develop their own proprietary tools. This trend is driven by the need for flexibility, scalability, and business continuity. Let’s explore why large corporations opt for in-house development and examine some real-world examples of proprietary tools that illustrate this choice.
Case Study 1: Yandex’s Internal Cloud
Problem Statement
Yandex’s internal cloud, known as Runtime Cloud (RTC), emerged in the early 2010s. At that time, the company realized that maintaining dedicated servers for each service was unsustainable. They needed a solution that could manage a large number of services while reducing costs and improving efficiency. Off-the-shelf cloud solutions like Kubernetes (k8s) weren’t viable because they were either too new or didn’t meet Yandex’s specific requirements.
Development Strategy
To create RTC, Yandex built a system to automate deployments, manage clusters, and ensure application isolation. Over time, this evolved into a comprehensive cloud infrastructure with custom solutions like Porto for containerization and Yandex Planner for resource management. These tools were designed specifically to meet Yandex’s unique needs, offering greater control and flexibility.
Case Study 2: Yandex’s Monorepository
Objective
Yandex’s monorepository, Arcadia, is designed to be a unified development ecosystem, offering version control, build systems, and CI/CD. It was created to facilitate code reuse and enable efficient distributed builds across Yandex’s many services. Existing solutions like SVN and Mercurial proved insufficient due to scaling issues and performance constraints.
In-House Approach
To address these limitations, Yandex developed a proprietary version control system called Arc. This system uses FUSE (Filesystem in User Space) and YDB, a scalable database. Additionally, Yandex created a custom build system, ya make, which offers faster performance compared to other open-source options like CMake or Bazel.
Common Reasons for In-House Development
Innovation and Scalability
Large tech companies often face unique challenges that require innovative solutions. As these companies grow, they encounter scalability issues that existing tools can’t always address. By building proprietary solutions, they can better manage their growth and remain flexible.
Inertia and Business Continuity
In-house tools provide greater control over critical processes, reducing the risk of vendor lock-in. However, developing custom tools can lead to inertia, where companies continue to rely on their proprietary systems even when standardized options become available. This inertia is often driven by the high cost of migrating from one solution to another and concerns about business continuity.
Migration Challenges
Switching from a proprietary solution to a standardized one can be complex and resource-intensive. This factor contributes to the persistence of in-house tools, as companies must weigh the costs and risks of migration against the potential benefits of adopting a standardized solution.
Conclusion
Big tech companies often choose to build their own tools for various reasons, including innovation, scalability, business continuity, and control. While standardized tools offer many benefits, they may not always align with the unique needs of large corporations. The cases of Yandex’s internal cloud and monorepository demonstrate the complexities and motivations behind creating in-house solutions. Ultimately, deciding whether to build or buy requires a thorough analysis of the organization’s context and a clear understanding of the associated costs and risks.
