Managing JavaScript dependencies


Hello everyone! My name is Slava Fomin, I am a leading developer at DomClick. Over the course of my 16-year practice, I have been in the forefront of the formation and development of JavaScript as a standard and ecosystem. In our company, we use JavaScript, first of all, for advanced front-end development and managed to try a fairly large number of different technologies, tools and approaches, fill a lot of cones. The result of this painstaking work was the most valuable experience that I want to share with you.

Initially, I wanted to talk about how we work on hundreds of packages using mono-repositories, but as a result I came to the conclusion that to go deeper into this rather complicated topic, it would be worthwhile to first explain simpler concepts. As a result, I decided to arrange all the material in the form of a blog with small posts in which I will expand on individual topics in more detail, smoothly leading the reader from simple to complex questions. I hope this will make the material useful not only to established professionals, but also to beginners.


Here is a list of topics I plan to cover on this blog:


  • What is a package, package manifest and dependencies.
  • How to properly describe dependencies for different types of projects.
  • How semver works and how to use version ranges correctly in the project manifest.
  • How installed dependencies can be represented in the file system, the pros and cons of different solutions.
  • How resolving searches works.
  • What are the tools for working with dependencies.
  • How to update dependencies correctly.
  • How to monitor security, track and prevent threats.
  • What are lock files for and how to use them correctly.
  • How can you efficiently work on hundreds of packages at the same time using mono-repositories and special tools.
  • What are phantom packets, where does the problem of duplicate packets come from and how can this be dealt with.
  • How to efficiently and safely use a package manager in the context of a CI/CD.
  • and more.

So, let's not lose time!


... We are like dwarfs sitting on the shoulders of giants; we see more and further than they, not because we have better eyesight, and not because they are taller than them, but because they raised us and increased our growth with their own greatness...

- Bernard of Chartres


I think few people will deny that the success that is now observed in the field of web development we were able to achieve largely thanks to all those people who created and maintain thousands of useful libraries and tools every day. Indeed, can you imagine the effective development of a modern web application without the use of frameworks, libraries and tools?


Despite the tremendous progress in fundamental web technologies, the creation of even a little bit complicated application would require quite a high qualification of the developers and writing a huge amount of basic code from scratch. But, thank God, there are such solutions as Angular, React, Express, Lodash, Webpack and many others, and we do not need to reinvent the wheel every time.


A bit of JavaScript history


We can recall the "wild times" when the code of popular libraries (such as jQuery) and their plug-ins needed to be downloaded directly from the official site and then unpacked from archives into the project directory. Of course, such libraries were updated in exactly the same way: manually. The assembly of such an application also required a manual and rather creative, unique approach. About the optimization of the assembly, I will not even mention.


To the great happiness that younger developers are unlikely to experience right now, these days of manual dependency management in the past. Now we have, although not the most advanced, but quite working tools that allow you to take control of dependency management even in fairly complex projects.


Node.js comes to the rescue


Modern web development is already completely impossible to imagine without Node.js , a technology that was originally developed for the server, but later became the platform for any JavaScript -projects, both front-end applications, and all kinds of tools, and with the popularization of SSR, the boundary between the environments began to completely disappear. Thus, the package manager for Node.js (Node Package Manager, or npm ), gradually became a universal package manager for all libraries and tools, written in JavaScript.


It is also worth noting that before the ESM JavaScript standard, in principle, the JavaScript language did not have the concept of modules and dependencies: all the code just loaded through the CDMY0CDMY tag in the browser and executed in one large global scope. For this reason, Node developers have implemented their own module format . It was based on the unofficial standard CommonJS (from the words “common/universal JavaScript,” or CJS), which later became factual standard in the industry. The Node dependency search algorithm (Node.js module resolution algorithm) itself has become the standard for representing packages in the file the project system, which is now used by all bootloaders and build tools.


Package all over the head


As mentioned above, Node.js introduced its own format for submitting and searching for dependencies, which is now de facto a common standard for JavaScript projects.


The system is based on the concept of a package: an npm package is the smallest unit for distributing JavaScript code. Any library or framework is represented as one or more related packages. Your application is also a package.


Before publication, a package is usually compiled and then loaded into a repository called the npm registry. The main use is the centralized official npm registry, which is publicly available on the domain registry.npmjs.org. However, the use of private closed npm registry is also common (we at DomKlik actively use this for internal packages). Other developers can install the published package as a dependency in their project by downloading it from the registry. This happens automatically with a package manager (like npm).


You can find the package you need or study their list on the npm official website .


The published package is physically a versioned archive with code that is ready for execution or implementation in another project. This code can be a library that you can include in your application (for example, lodash), or a complete program that can be directly called on your computer (for example, webpack).


In this case, code from one package can access code from another. In this case, they say that one package depends on another. Each package can have many dependencies, which, in turn, can also have their own dependencies. Thus, the links between all packages form a dependency tree :


ITKarma picture


The image shows the result of the CDMY1CDMY command - the project dependency tree in which only two packages are installed: the Express HTTP server (with many child dependencies) and the Lodash library (without dependencies). Note that the same CDMY2CDMY dependency occurs 4 times in different parts of the tree. The inscription CDMY3CDMY means that npm detected duplicate dependencies and installed the package only once (we will talk more about duplication in the following posts).


Since the Node ecosystem preaches the Unix philosophy , when one package has to solve its own narrow task and do it well, then the number of dependencies in the average project can be very large and easily exceeds several hundred.This leads to the fact that the dependency tree grows very much both in width and in depth. Probably only the lazy one did not joke about the size of the CDMY4CDMY directory in which all these dependencies are installed. Often, outsiders criticize JavaScript for this:


ITKarma picture


Package manifest


What is a package and how can we create it? In fact, a package can be any directory containing a special manifest file: package.json . It can contain a lot of useful package information, such as:


  • name, version and description,
  • type of license,
  • home page URL, git repository URL, page URL for bug reporting,
  • names and contacts of authors and maintainers,
  • keywords so the package can be found
  • file paths to library code or executable files,
  • list of dependencies,
  • auxiliary local commands (scripts) for working on the package,
  • et al. (see the complete list ).

Example manifest package.json.


Description of package dependencies


The package manifest contains a number of optional fields that allow you to specify a list of dependencies:


  • CDMY5CDMY,
  • CDMY6CDMY,
  • CDMY7CDMY,
  • CDMY8CDMY.

Each of these fields is a JSON object, where the key is the name of the package, and the value is the range of versions that are supported in your project.


Example :


{ … "dependencies": { "lodash": "^4.17.15", "chalk": "~2.3", "debug": ">2 <4", }, … } 

Let's look at the purpose of each field individually.


dependencies


The CDMY9CDMY field defines the list of dependencies, without which the code of your project will not be able to work correctly. This is the main and main list of dependencies for libraries and programs on Node.js. If your code has imports of some third-party dependencies, for example CDMY10CDMY, then this dependency should be written in the CDMY11CDMY field. Its absence will lead to the fact that when executed, your program will crash because the desired dependency will not be found.


devDependencies


The CDMY12CDMY field allows you to specify a list of dependencies that are necessary only at the package development stage, but not to run its code in runtime. This includes all sorts of development and assembly tools, such as typescript, webpack, eslint, and others. If your package will be installed as a dependency for another package, then the dependencies from this list will not be installed .


peerDependencies


The CDMY13CDMY field plays a special role in developing support packages for some tools and frameworks. For example, if you are writing a plugin for Webpack, then in the CDMY14CDMY field you can specify the version of webpack that your plugin supports.


If your package is installed in a project where the specified dependency is missing, or the condition for the version is not satisfied, the package manager will inform the developer about this.


When you install your package, the dependencies from the CDMY15CDMY field are not automatically installed , the developer himself must install them in his parent project by writing down his package in the manifest. This also ensures that the project will use only one version of the dependency without duplicates. Indeed, if several different versions of, say, Webpack are installed in the project, this can lead to serious conflicts.


optionalDependencies


And the last field CDMY16CDMY allows you to specify non-critical dependencies, without which your application can still do its job. If the package manager cannot find or install such a dependency, it will simply ignore it and not cause an error.


It is important to understand that your code must correctly respond to the absence of such dependencies, for example, using the CDMY17CDMY bundle.


Dependencies in front-end projects


Above, we looked at four ways to define various dependencies for your project. However, do not forget that this system was originally invented for applications and libraries on Node.js that run directly on the user's machine, and not in a special sandbox, which is the browser. Thus, the standard does not take into account the features of front-end application development in any way.


If someone tells you that one way to determine npm dependencies for front-end applications is correct and the other is not, then do not believe: there is no “correct” method, because this use case is simply not taken into account in node and npm.


However, for the convenience of working on front-end applications, I can offer you a time-tested and dependency-defining format. But first, let's try to figure out how the front-end project differs from the project on Node.js.


Typically, the ultimate goal of a Node.js developer is to publish the package he created in the npm registry, and from there this package is downloaded, installed and used by the user as ready-made software or as a library as part of a more complex product. In this case, dependencies from the CDMY18CDMY field in the package manifest are installed in the end-user project.


In the case of the front-end application, it is not published in the npm registry, but is assembled as an independent artifact (static) and uploaded, for example, to CDN. In fact, npm in front-end projects is used only to install third-party dependencies. For this reason, it is recommended that you use the CDMY19CDMY option in the manifest of such a project, which ensures that application files are not accidentally sent to the public npm-registry. The name and version of the application package itself do not make sense, since the "outside" are not used anywhere.


This feature of front-end applications allows us to use the CDMY20CDMY field not entirely for its intended purpose, but as a category in order to divide the list of dependencies into two parts: in the CDMY21CDMY field you write a list of direct dependencies that are used in the application code, for example, CDMY22CDMY, CDMY23CDMY, CDMY24CDMY, etc., and in the CDMY25CDMY field, dependencies that are needed to develop and build the application: CDMY26CDMY, CDMY27CDMY, declarations from CDMY28CDMY packages, etc.


Wait, but this is no different from how dependencies are written for packages on Node.js! Yes, however, some particularly pedantic developers may say that since third-party dependencies are combined into the application bundle during assembly and are not actually imported in runtime, they should be in the CDMY29CDMY field. Now you can reasonably defend a more practical approach.


Semantic versioning


The npm ecosystem adopted the standard for package versioning semver (from the words Semantic Versioning (semantic versioning)).


The essence of the standard is that the package version consists of three numbers: the major version, the minor version, and the patch version:


ITKarma picture


For example: 3.12.1.


This type of versioning is called semantic because each version number, or rather, an increase in the number, has a certain meaning.


An increase in the patch version means that minor corrections or improvements have been made to the package that do not add new functionality and do not violate backward compatibility.


Increasing the minor version means that new functionality has been added to the package, but compatibility has been preserved.


The increase in the major version means that the package has made major changes to the API, which led to a loss of backward compatibility and the user of the package may need to make changes to his code in order to upgrade to the new version. Such changes and the migration order to the new version can usually be found in the CHANGELOG file at the root of the package.


Volatile versions


Package versions prior to 1.0.0, for example, 0.0.3 or 0.1.2, semver also has a certain meaning: such versions are considered unstable and an increase in the first nonzero version number should be regarded as a change with a potential violation of backward compatibility.


To be continued


We looked at the very basics of dependency management in JavaScript: we learned what a package is, how it is defined, and how dependencies are defined. In the next post, we will take a closer look at how semver works in practice, how to correctly prescribe version ranges and update dependencies.


Stay tuned!

.

Source