Source maps in Node.js
Contributed by Benjamin Coe, who works on JavaScript client libraries at Google, is a collaborator on Node.js, and was the third engineer at npm, Inc.
Of the 21,717 respondents to the 2019 State of JavaScript Survey, ~60% said that they spend time working in an alternate flavor of JavaScript, this is up from ~21% in 2016. Increasingly, when someone writes JavaScript, they’re actually writing an abstraction that compiles to JavaScript.
These abstractions offer a variety of benefits: for instance the type safety introduced by Flow and TypeScript, or the functional programming paradigm introduced by ClojureScript. However, we are also faced with a challenge. Node.js, developed in 2009, didn’t anticipate the modern world of transpilers. And so alternate flavors of JavaScript present a disadvantage in terms of observability. Take the following TypeScript code:
enum HttpStatusCode {
NOT_FOUND = 404,
ERROR = 500
}
class MyHttpError extends Error {
code: HttpStatusCode;
constructor(msg: string, code: HttpStatusCode) {
super(msg);
Error.captureStackTrace(this, MyHttpError);
this.code = code;
}
}
throw new MyHttpError('not found', HttpStatusCode.NOT_FOUND);
When compiled to JavaScript, the above code ends up looking like this:
var __extends = (this && this.__extends) || (function () {
var extendStatics = function (d, b) {
extendStatics = Object.setPrototypeOf ||
({ __proto__: [] } instanceof Array && function (d, b) { d.__proto__ = b; }) ||
function (d, b) { for (var p in b) if (b.hasOwnProperty(p)) d[p] = b[p]; };
return extendStatics(d, b);
};
return function (d, b) {
extendStatics(d, b);
function __() { this.constructor = d; }
d.prototype = b === null ? Object.create(b) : (__.prototype = b.prototype, new __());
};
})();
var HttpStatusCode;
(function (HttpStatusCode) {
HttpStatusCode[HttpStatusCode["NOT_FOUND"] = 404] = "NOT_FOUND";
HttpStatusCode[HttpStatusCode["ERROR"] = 500] = "ERROR";
})(HttpStatusCode || (HttpStatusCode = {}));
var MyHttpError = /** @class */ (function (_super) {
__extends(MyHttpError, _super);
function MyHttpError(msg, code) {
var _this = _super.call(this, msg) || this;
Error.captureStackTrace(_this, MyHttpError);
_this.code = code;
return _this;
}
return MyHttpError;
}(Error));
throw new MyHttpError('not found', HttpStatusCode.NOT_FOUND);
//# sourceMappingURL=test.js.map
The generated source looks a lot different than the original source. Take the statement throw new MyHttpError
. In the original source this error is thrown on line 13, but in the generated source it's thrown on line 29. These differences make debugging difficult — the stack trace displayed in the code being run doesn’t match the code being written.
The increasing popularity of alternate flavors of JavaScript and the observability challenges described above are the motivation for the recent work in Node.js related to source maps (node#28960, node#31143, node#31132).
The Source Map V3 specification
Source maps provide a method for translating from generated source back to the original, via meta-information attached to the generated source; In JavaScript and CSS Source Map Revision 3 has become the de facto standard. Here are some key facts about the specification:
- V3 of the spec was created as a joint effort by engineers John Lenz at Google and Nick Fitzgerald at Mozilla.
- The format trades simplicity and flexibility for a reduced overall size, making it more practical to use source maps in real-world environments, e.g., loading them over a network when DevTools are opened.
- The format allows for 1 to 1 mappings, e.g., a
.ts
file mapping to its compiled.js
version, and for 1 to many mappings, e.g., many sources being minified into a single source file. - Source maps are embedded in the generated source using a special comment. These comments may contain the entire source map, using a Data URI, or may reference an external URL or file.
Looking back at the TypeScript example earlier in this article, note the line at the end of the generated source:
//# sourceMappingURL=test.js.map
This special comment indicates that the source map can be found in the file test.js.map
, which is in the same folder as the generated source code. Most tools that transpile JavaScript, such as TypeScript, provide an option to generate source maps.
Adding support for the Source Map V3 format to Node.js was an important step towards better supporting the alternate flavors of JavaScript that are being written today.
Caching source maps for code coverage
In v12.11.0
, the behavior of the environment variable NODE_V8_COVERAGE
was updated, such that when a program is run with this variable set, and a require
or import
observes a special source map comment, the corresponding source map is loaded and cached.
As the variable name suggests, source map support was initially added to address a problem with native V8 code coverage; tools like ts-node and nyc insert source maps during run time, and these source maps were not available after a program finished execution. Due to this, accurate coverage reports could not be provided. By caching source maps during runtime, and outputting the information with coverage reports, Node.js’ native code coverage could be made to provide accurate reports for tools like ts-node.
Along with addressing bugs with code coverage, this work created a foundation for adding further source map support to Node.js.
Applying source maps to stack traces
In v12.12.0
, Node.js introduced the flag --enable-source-maps
. When a program is run with this flag set, source maps will be cached, and used to provide an actionable stack trace when an exception occurs. Let's look at what happens when we execute the TypeScript example in the first section of this article:
Error: not found
at Object.<anonymous> (/Users/bencoe/oss/source-map-testing/test.js:29:7)
-> /Users/bencoe/oss/source-map-testing/test.ts:13:7
at Module._compile (internal/modules/cjs/loader.js:1151:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
at Module.load (internal/modules/cjs/loader.js:1000:32)
at Function.Module._load (internal/modules/cjs/loader.js:893:16)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
at internal/main/run_main_module.js:17:47 {
code: 404
}
The stack trace includes both the position of the exception in the generated code, /Users/bencoe/oss/source-map-testing/example.js:29:7
, and the position of the exception in the original source, -> /Users/bencoe/oss/source-map-testing/example.ts:13:7
.
This makes it much easier to debug exceptions in alternate flavors of JavaScript, such as ClojureScript and TypeScript, without introducing a single dependency to your application.
How can I use this feature?
You can start using Node.js’ source map functionality today: make sure you have an up-to-date version of Node.js installed, and run your programs with the flag --enable-source-maps
.
Here are a couple interesting ways you could set this flag:
Turning on source-map support on for a single run of an npm script, using NODE_OPTIONS
:
NODE_OPTIONS=--enable-source-maps npm test
Configuring mocha to set the flag, using .mocharc.json
:
{
"enable-source-maps": true
}
What’s next
Applying source maps more extensively in Node.js
There are still parts of the Node.js runtime that don’t take into account source-maps, for instance, the logic applied to show an error in its original context:
/Users/bencoe/oss/source-map-testing/test.js:30
throw new MyHttpError('not found', HttpStatusCode.NOT_FOUND);
^
We would gradually like to make an effort to address these gaps in the source map implementation.
Interested in contributing to the Node.js project? Addressing some of these edge-cases where source maps aren’t yet applied is a great place to start.
Applying source maps to userland tooling
In v13.7.0
a public API was introduced for interacting with source maps. It's my hope that this can be leveraged by userland tooling, like stack-trace and winston, to provide better support for transpiled code.
I’m excited to share the work we’ve been doing to help Node.js better support the many exciting flavors of JavaScript being written today. Try it out!