Software Development Tools: A Comprehensive Overview

When I learn a programming language, one of the first things I try to understand is how to transform written code into a deployable (installable) and runnable artifact. I start by determining the programming language type, specifically whether a compilation step is required and if an interpreter is required at run time. Then, I look into the programming tools or toolchain necessary to build and package any application I develop.

This article discusses the different types of tools needed to manage software build and packaging. First, we explain the common methods for setting up the toolchain. Then, we describe the toolchain tools.

​

Getting the Toolchain Ready

Before building the code, you need to have the required programming tools available. You can achieve this in one of the following ways:

• Manual Installation: Install them using the system package manager or from the source.
• Automated Installation: Use a toolchain management tool or an installation script for automated installation, if available.
• Development Containers: Use a pre-built virtual environment container (e.g., Docker), called development container, based on an image, if available.

For programming languages that require an interpreter, it will also be made available in this manner, both for the build stage and the run time.

You can usually use the whole toolchain through the programming language’s Integrated Development Environment (IDE) software or a Continuous Integration (CI) pipeline.

​

Components of the Toolchain

Let’s explore the different types of tools and how they are typically organized within a software build pipeline.

​

Primary Tools

This category of tools directly processes the code. They are used to check the source code, generate code documentation, automatically generate source code from Interface Description Language (IDL) and macro languages, compile the code (if needed), test the code, package the software, and upload it to the package repository (if needed).

​

1. Code Editing

Many tools can help developers write code more efficiently. These include tools for autocompletion, navigation (within and across source files), and syntax highlighting. These tools use lightweight analysis and are often built into text editors designed for specific programming languages.

Many modern code editors use the Language Server Protocol (LSP) to interact with these code writing productivity-boosting tools. They also use it to interact with source conformity analysis tools.

​

2. Source Conformity Analysis

Source conformity analysis tools are primarily used to:

• Check that the source artifacts follow best practices and are free of trivial errors. These specific tools are usually referred to as linters.
• Ensure that the source artifacts follows the project’s formatting standards. These specific tools are usually referred to as formatters.

Some source conformity tools fulfill both the linter and formatter functionalities.

​

Linter

Linters parse the source code, perform various checks, and report issues for developers to fix. Detecting errors early helps prevent issues during compilation (if applicable), testing, or runtime, thus improving the software development life cycle (SDLC). Here are some common types of checks performed by linters:

• Syntax correctness checks: Ensure there are no syntax errors in the source artifact. This is particularly useful for programming languages that lack a build-time compilation step. It is also essential for configuration files.

• Convention checks: Verify adherence to project guidelines regarding coding style, variable naming, comment requirements, code complexity, preferred syntax, code smells and more. They play a crucial role in maintaining consistency across a project and making the software easy to maintain. Convention requirements do not affect the semantic but only the syntax, and potentially the program performance.

• Common semantic errors and security checks: These are checks for potential semantic errors and vulnerabilities such as uninitialized variables, memory leaks, null pointer dereferences, etc.

Some linters can provide automatically generated fixes for the issues they detect.

​

Formatter

Formatters automatically transform the source artifacts to follow the project’s style guidelines. The style requirements implemented by formatters include: indentation size and type (tabs vs. spaces), line breaks, maximum code line length, use of curly braces and their placement (when supported), spacing before parentheses (when supported), and more.

Formatters check the sources’ style and transform it to follow the style guidelines. This transformation does not affect the semantics of the code.

​

3. Automated Documentation Generation

Particularly useful for Application Programming Interface (API) documentation, automated documentation generation tools create visually appealing documentation by extracting the documentation from source artifacts. The extracted documentation is often rendered into HTML format or other documentation formats such as reStructuredText or Markdown.

Source documentation tools are often language-specific, meaning they usually support one or a few programming languages or domain-specific languages (DSL).

These tools also use dedicated documentation files to generate project documentation. In fact, some documentation generation tools only use dedicated documentation files (such as reStructuredText or Markdown) to generate a documentation webpage. This type of documentation generation tool should be used after generating documentation files from the source artifacts with language-specific source documentation generation tools. An example of such a tool is Mkdocs.

Documentation generation tools are configurable, allowing a configuration file to define how the documentation should be generated. Examples of configuration options include:

• Source code or dedicated documentation files to exclude during generation.
• Final documentation format.
• Style and theme to use for the HTML documentation.

​

4. Automated Source Code Generation

There are scenarios in software engineering where source code is automatically generated and used to build or execute a program. The most common scenarios involve the use of macros and Interface description languages (IDL).

An example of using macro is a system tool with a component that makes some system calls that vary across different platforms. A macro processor such as M4 can generate the correct calls at build time based on the selected platform. In this case, the various platform-specific calls are developed using macros.

An example of using IDLs is a microservice that accesses a server through gRPC, where the server’s services are defined using the Protocol Buffers IDL. The microservice code accesses the server’s services through stub code generated from the Protocol Buffers files.

​

5. Compilation (when needed)

Some programming languages require that the source code is compiled into machine code or interpreted code before runtime. This is the case for native languages such as C, C++, Rust, Go, as well as some interpreted languages such as Java and Typescript. The compilation process involves several tools, depending on the target output:

• Compilation into machine code: The compiler transforms source code files into assembly code. The assembler then converts the assembly code into machine code. For a library, the archiver bundles the machine code files into a single archive. For a standalone executable, the linker combines the machine code files into one, resolving references and linking external elements.

• Compilation into interpreted code: The compiler converts the source code files into interpreted code. The archiver then bundles these interpreted code files into a single archive.

For programming languages that do not require compilation, the source code files themselves are executable.

​

6. Testing

It is important to check that a program works as expected, meaning it produces the correct outputs for given inputs. For example, consider a program that:

• Reads an integer from the standard input.
• Prints “Is prime” if the number is prime; otherwise, prints “Is not prime”.

If we run the program with the input 3 and it prints “Is not prime”, we know there’s an error that needs fixing.

Testing tools help automate this process by allowing us to easily specify inputs and expected outputs, execute the program, and report any discrepancies. If the program’s output doesn’t match the expected result, it indicates a bug that needs to be addressed.

After identifying errors using testing tools, the program might need modification to fix them, and the development process returns to the code editing phase.

​

7. Packaging Artifacts

When we are satisfied with the built executable code, we may package it for easy deployment or to make it available on a package repository.

Packages may be created in the native form of the programming language package manager or as container images of the program, such as those created with Docker. The latter is very common for the microservices architecture.

Regarding Docker images, the Docker CLI tool can be used to create images using the docker build ... command.

​

8. Uploading Packages

Once packages are created, they often need to be uploaded to a public or private package repository. These repositories provide tools, called package publishers, to upload packages.

For example, with Docker images, the Docker CLI tool can be used to publish images using the docker push ... command.

​

Intermediary Tools

The category “intermediary tools” consists of tools that are not directly linked with code editing and building. This includes tools for dependency management, build orchestration, and source code management (version control) systems.

​

1. Dependencies Management

Software engineers often use existing programs, known as dependencies, to avoid “reinventing the wheel”. It is crucial to keep track of the dependencies used by a project and their versions. It is also important to quickly and automatically make them available.

This dependency management task is often handled by tools called package managers or dependency managers. Modern package managers use configuration files, often stored within the project repository, to ensure dependencies are available during the build.

​

2. Build Orchestration

The build process involves multiple steps and tools. To simplify and automate this process, a build orchestrator or build tool is used. This tool is controlled through configuration files that describe, in a declarative or imperative manner, the steps of the build process.

​

3. Source Code Managers

Using source code managers, also known as version control tools, such as git, is essential for managing software source code repositories. Source code management tools allow developers to:

• Store their software source code in remote repositories.
• Keep a history of software versions and switch between them.
• Develop different software features in parallel and merge them once completed.
• And more.

​

Things to Look For in a Code Repository

A source code repository will often contain the following elements:

• Developed source code files
• DSL source files
• Test code and data files
• Dedicated documentation files
• (Optional) Software runtime configuration files
• Source code manager configuration files (e.g., .gitignore for Git)
• build orchestrator tool configuration files (e.g. Makefile)
• Package manager configuration files
• Text editor configuration files
• Linter configuration files
• Code formatter configuration files
• testing tool configuration files
• Documentation generator configuration files
• DSL-based code generator configuration files
• Compiler infrastructure configuration files
• Package generator and publisher configuration files