Demystifying Amazon’s Distinctive Software Development System

Author | Gergely Orosz

Translator | Hirakawa

Planning | Tina

This article was originally published on Gergely Orosz’s personal blog.

Amazon has a ton of internal systems. As a software engineer and engineering manager, the following are worth knowing.

When joining as an SDE (software development engineer) or SDM (software development manager), you have to learn to use Amazon’s custom tech stack, which is surprisingly different from the tech stack used by AWS customers. Below are the systems you may encounter.

Internal systems closely related to SDE

Code: Code Search and VCS (Git).

Crux: Amazon’s code review system.

Brazil: Amazon’s build system. Check out this article detailing Brazil, which is outdated but still relevant.

Sage: “Stack Overflow” within Amazon.

Amazon’s internal wiki system: This system has some nice features, like it’s easy to embed pictures from AWS Cloudwatch or previous systems ( PMET – Performance Metrics) on the page.

NAWS (Native AWS): The “modern” technology stack using the current AWS.

MAWS (Move to AWS): Legacy legacy AWS system.

• A lot of systems are moving away from this, especially in retail.

• MAWS requires services to be launched on EC2 instances via a system called Apollo, which is largely deprecated in NAWS (you should use Lambda or ECS, or raw EC2 if absolutely necessary).

Isengard/Conduit: AWS account management, all systems are isolated, which ensures that each region, service, stage (stage) has a unique AWS account.

Last Week in AWS editor Corey Quinn described the service as “the AWS service he hates the most.” He wrote:

The fact that this is an internal system for provisioning AWS accounts means that the way the AWS engineers who built AWS manage AWS accounts is insulated from the way the rest of the world manages AWS accounts. They don’t need to do it the same way as AWS Organizations, Landing Zones, Control Tower, or AWS SSO. This is the root cause of my dissatisfaction with this service.

Pipelines: CI/CD system that supports multi-stage deployment (up to 4 stages: beta, gamma, prod, local). One AWS engineer described it this way:

At Amazon, pipelines are one of the best examples of “making easy things difficult and hard things possible.” Services deployed to stage 3-4 (beta, gamma, and prod across different regions) presumably don’t care about pipelines. And like most AWS services, services with hundreds of deployment units in the pipeline are very happy with it.

LPT: Dynamic Pipeline Template. This is a Ruby library that generates CloudFormation or CodeDeploy templates that also define pipelines, Isengard accounts, and other scaffolding. Typically, each service has an LPT package to create the required resources.

AWS CDK: Amazon is pushing for it to replace LPT, but as of early 2022, it’s still a less mature system than LPT. Most teams are taking action, and some say they especially like the TypeScript support it provides.

2PR: Second-person approval systems for sensitive operations, such as Isengard and SSH login systems. If the required approval is not obtained when accessing the system, a team violation ticket is automatically created, which can be escalated to management.

system at the organizational level

AWS Chime: Formerly Amazon’s chat and video calling app. Now, Amazon uses Slack for chat, but AWS Chime is still used for video calls, including phone interviews.

Kingpin: Team, organization, and Amazon company-wide object tracking system.

Accolades: A tool for complimenting employees through reviews, and provides a way to easily CC managers and others.

Connections: Preinstalled on company laptops. It starts the day with a simple question like “How do you feel about your manager” or “How much does your team value Operational Excellence (OE)?” and asks you to rate it out of 5 rating. Everyone in the company sees the same problems every day.

Forte : Amazon’s company-wide performance feedback process that runs from late December to the end of January. Employees use Forte tools to ask for feedback from their peers and those they work with. Short feedback, 60 words or less.

Aggregated, anonymous results are uploaded to management, whose goal is to improve their Connections score over time. These results are shared with the team, the team provides feedback, and their manager takes action accordingly.

Programming language

Most services are written in Java. However, teams are autonomous, and they can choose whatever language and framework they want to use. While Java is predominant, a number of other languages ​​are also used in these services.

​​

The text and pictures in this article are from InfoQ