RexReplace is a versatile tool to search and replace text in files from the command line. It's inspired by how developers often need to do quick fixes or one-liners for build scripts.
Key features:
- Easy and intuitive notation makes you trust what you are doing
- Replacement can be javascript code - giving you Turing complete flexibility
- Pinpoint the exact files with glob notation (
docs/*.mdrepresents each markdown file indocs/) powered by fast-glob for optimal performance - No more brute-forcing the right combination of
find,cat,sed,tr, andawkto replace a text pattern in the right files
To use RexReplace from your command line
> npm install -g rexreplaceTo use RexReplace from an npm build script:
> yarn add rexreplace --dev
# or
> npm install rexreplace --save-devTo use Bun as the runtime:
- Use the command
rexreplace-bun. It is installed together withrexreplaceandrr. - Or install the package
rexreplace-bun. Then all three commands use Bun. Do not install both packages globally.
> bun install -g rexreplace-bunWith Bun, the version ends with +bun and the regex engine is JavaScriptCore (Node has V8).
Let 'foobar' become 'xxxbar' in myfile.md
> rexreplace 'Foo' 'xxx' myfile.mdHard for your fingers to type on your keyboard? We got you covered with the rr alias for rexreplace
> rr Foo xxx myfile.mdLet all markdown files in the docs/ dir get headlines moved one level deeper
> rexreplace '^#' '##' docs/*.mdFix a spelling error in all javascript and typescript files in the folders src/ and test/ recursively.
> rexreplace 'foubar' 'foobar' '{src,test}/**/*.{js,ts}'Let the version number from package.json get into your distribution js files (use the string VERSION in your source files).
> rexreplace 'VERSION' 'require("./package.json").version' -j dist/*.jsRequire has been given the alias r and both are expanded to understand relative paths even without ./ prepended. As the file extension is not needed either, you will get the same result writing:
> rexreplace 'VERSION' 'r("package").version' -j dist/*.jsLonger code can live in its own file. Let the file export a function, and give it the match:
// fix.js
module.exports = match => match.toUpperCase();> rexreplace 'foo' 'r("fix")($0)' --js myfile.mdLet 'foobar' become 'barfoo' in myfile.md
> rexreplace '(foo)(.*)' '$2$1' myfile.mdRexReplace defaults to treating € as an alias for $ so the following will do the same as the previous example
> rexreplace '(foo)(.*)' '€2€1' myfile.mdLet foo[bar] become foo.0.[bar] in myfile.md
> rexreplace '\[' '.0.[' myfile.mdThe [ as a literal char must be escaped according to regex. If you run the command as a parameter (this could be from a script in package.json) you need to escape the escape:
"scripts":{
"fix": "rexreplace '\\[' '.0.[' myfile.md"
}RexReplace defaults to treating § as an alias for \ so the following give same result:
> rexreplace '§[' '.0.[' myfile.md"scripts":{
"fix": "rexreplace '§[' '.0.[' myfile.md"
}Per file info
Add creation time, name of the file and human readable file size as the first line in each file in test-run recursively.
> rexreplace '^' 'ctime_ + name_ + size + nl' -j -M 'dist/**/*.*'List changed files
The path of each file that changed is printed to stdout, one name on each line.
> rexreplace 'foo' 'bar' '**/*.md' | xargs git addUse --null-out (or --null) to separate names with \0 instead of a new line. Can be helpful when chaining data to avoid problems with space and newlins.
> rexreplace 'foo' 'bar' '**/*.md' --null-out | xargs -0 git addPipe the names into the next run with --glob-pipe, so it only works on the files the first one changed:
> rexreplace 'foo' 'bar' '**/*.md' --null-out | rexreplace 'baz' 'qux' --glob-pipe --null-inNull bytes are just chars in the search, so null-separated data can be made readable by turning them into line breaks:
> find . -name '*.md' -print0 | rexreplace '\x00' '\n'Matching pairs
Let both "foo 'is' bar" and ' foo "was" bar' become » foo ... bar « independent of using ' or "
> rexreplace '(['"])([^\1]+)\1' '» $2 «' myfile.mdIf you provide no files/globs as parameters, the piped data is the text to search, and the result is printed just as it came in, with nothing added at the end.
> cat myfile.md | rexreplace 'foo' 'bar'Use the piped data as replacement with the -R flag. The -T flag trims the input.
> git rev-parse --short HEAD | rexreplace 'COMMIT' _ -R -T dist/*.jsUse --js-pipe to access the pipe data in your javascript code.
> git log -1 --format=%an | rexreplace 'AUTHOR' 'pipe.trim().toUpperCase()' --js-pipe docs/*.mdWith --replacement-file the replacement is the path to a file, and the content of the file goes in. --trim-file trims the content, like the line break at the end of the file.
> rexreplace '\{\{footer\}\}' partials/footer.html 'site/*.html' --replacement-file --trim-fileWith --js the code gives the path, so each {{name}} gets the content of partials/name.html:
> rexreplace '\{\{(\w+)\}\}' '"partials/" + $1 + ".html"' 'site/*.html' --js --replacement-file --trim-fileWith --rename the search and replace is done on the name of each file, and the file is renamed to the result. The folder and the content are not touched.
> rexreplace '\.markdown$' '.md' '**/*.markdown' --rename
docs/intro.md
docs/setup.mdWith --js the name is the text:
> rexreplace '^.*$' 'text.toLowerCase()' 'docs/*' --rename --js
docs/readme.mdWith --rename-path it is done on the whole path, so a / in the replacement moves the file. Folders that are not there are made.
> rexreplace '^' 'archive/' '*.log' --rename-path
archive/a.log
archive/b.logNumber the files with $n in the code. $n00 has three digits, and --sort decides the order:
> rexreplace '^.*$' '"photo-" + $n00 + ext' '*.jpg' --rename --js --sort modified
photo-001.jpg
photo-002.jpgSee what would happen first with --simulate. Each old name is printed (to stderr) just before its new name (to stdout), and nothing is renamed.
> rexreplace '\.markdown$' '.md' '**/*.markdown' --rename --simulate
docs/intro.markdown
docs/intro.md
docs/setup.markdown
docs/setup.mdIf planned renames clash, the process is halted and no change is applied to any file.
> rexreplace '^[ab]' 'c' a.md b.md --rename
❌ Renames to the same path are not allowed.
From: a.md
From: b.md
To: c.md
❌ Planned renames clash, so no change is applied to any file
No further processing.> rexreplace pattern replacement [fileGlob|option]+| Flag | Effect |
|---|---|
-v |
--version Print rexreplace version (can be given as only argument) [boolean] |
-V |
--verbose More chatty output [boolean] |
-L |
--literal Literal string search (no regex used when searching) [boolean] |
-l |
--literal-replacement No "smart" values in the replacement, so you can have text like $1 and $& without escaping them. [boolean] |
--replacement-file The replacement is the path to a file, and the content of the file is used as the replacement. With --js the code gives the path. [boolean] |
|
-I |
--void-ignore-case Void case insensitive search. Applies to the pattern and to all globs. [boolean] |
-G |
--void-global Void global search (stop looking after the first match). [boolean] |
-s |
--dot-all Have . also match newline. [boolean] |
-M |
--void-multiline Void multiline search pattern. Makes ^ and $ match start/end of whole content rather than each line. [boolean] |
-u |
--unicode Treat pattern as a sequence of unicode code points. [boolean] |
-e |
--encoding Encoding of files/piped data. [default: "utf8"] |
-E |
--engine What regex engine to use. default is the engine of the runtime: V8 with Node and JSC (JavaScriptCore) with Bun. The engine of the other runtime can not be picked. [choices: "default", "V8", "JSC"] [default: "default"] |
-q |
--quiet Hide file names and warnings. Errors are still shown, and output from flags like --output and --output-match is still shown. [boolean] |
-Q |
--quiet-total Never display errors or info [boolean] |
-f |
--full-path Print the full path of the named files instead of the relative path. [boolean] |
--name-all Print the names of all files found by the globs, also the ones where nothing was changed. [boolean] |
|
--name-delimiter String printed between the names of the files. Default is a line break, unless the name already ends with one. [string] |
|
--content-delimiter String printed between the content of each file for --output, or between the matches for --output-match. Default is a line break, unless the content already ends with one. [string] |
|
--delimiter Sets --name-delimiter and --content-delimiter to the same value. Supports escape sequences like \t, \n and the € and § aliases. [string] |
|
--null-out, --null Shorthand for --delimiter '\0'. Useful for piping content with spaces or line breaks (like for xargs -0). The specific delimiter values win over --delimiter, and --delimiter wins over --null-out. [boolean] |
|
--name-delimiter-tail Print the name delimiter after the last name. Defaults to true. Use --no-name-delimiter-tail or --name-delimiter-tail=false to not print the delimiter after the last name. [boolean] |
|
--content-delimiter-tail Print the content delimiter after the last content. Defaults to true, but to false when piped data is printed, so it is left as it came in. Matches from --output-match always get it, from a file or a pipe. [boolean] |
|
--tail Shorthand for setting both --name-delimiter-tail and --content-delimiter-tail. Use --no-tail to not print the delimiter after the last name and the last content. The specific tail flags win over --tail. [boolean] |
|
--nl Let what is printed to stdout and stderr (the names, or the content from -o, -m or piped data) always end with a new line. No extra new line if it already ends with a new line, or if nothing was printed. [boolean] |
|
-H |
--halt, --bail Halt on first error [boolean] [default: false] |
-c |
--status-compact As the last thing before ending any execution a status report will be printed as one line of JSON. Will print even in combination with --quiet-total. Will incude extra info about the inner workings of the engine in combination with the --verbose flag. [boolean] |
--status Same as --status-compact, but always starting with a \n followed by pretty printed JSON. [boolean] |
|
-d |
--debug Only print a status report, with what went into the engine and its steps. The result of --output and --output-match will still printed. [boolean] |
| -€ | --void-euro Void having € as alias for $. The alias is used in all inputs, but never in globs, file names or piped data. [boolean] |
| -§ | --void-section Void having § as alias for \. The alias is used in all inputs, but never in globs, file names or piped data. [boolean] |
| -o | --output Output the final result instead of saving to file. Will also output content even if no replacement has taken place. [boolean] |
| -A | --void-async Handle files in a synchronous flow. Good to limit memory usage when handling large files. [boolean] |
| -B | --void-backup Avoid temporary backing up file. Works async (independent of -A flag) and will speed up things but at one point data lives only in memory, and you will lose the content if the process is interrupted. [boolean] |
| -b | --keep-backup Keep a backup file of the original content. With --rename the old file is kept. [boolean] |
| -S | --simulate Execute everything during a run except changes RexReplace would normally make to the file system. Code from --js still runs, and can change data even with --simulate. [boolean] |
| | --sort Process files sync in a sorted order chosen from 6 values: natural (numbers compared as numbers and case insensitive), ascii (by char code, so B comes before a), modified, created, size, or none (the order of the input: files first, then files found by globs). Files sorted the same (like the same size) are ordered by ascii sort. Without --sort files are processed asynchronously, so output order is not guaranteed, but the processing starts in natural order. [string] [choices: "natural", "ascii", "modified", "created", "size", "none"] |
| | --sort-reverse Turn the order of the files around. [boolean] |
| -F | --expect-files Check that at least one file is found. The exit code is 1 if not. [boolean] |
| -z | --expect-diff Check that at least one file is changed. The exit code is 1 if not. [boolean] |
| -Z | --expect-zero-diff Check that no file is changed. The exit code is 1 if one is. [boolean] |
| | --expect-diff-all Check that all files found are changed. The exit code is 1 if not. [boolean] |
| -r | --rename Search and replace in the name of each file, and rename the file to the result. The folder and the content are not touched. If planned renames clash the process is halted and no change is applied to any file. The process will halt before renaming begins if old or new names contain ASCII control characters, end on a space or a dot, or consist only of spaces and dots. [boolean] |
| | --rename-path Like --rename, but on the whole path of each file, so a / in the replacement moves the file. Folders that are not there are made. [boolean] |
| | --rename-force With --rename or --rename-path, write over a file that is there already. [boolean] |
| -m | --output-match Output each match on a new line. Will not replace any content but you still need to provide a dummy value (like _) as replacement parameter. The replacement is not used, so --js code is not run. If search pattern does not contain matching groups the full match will be outputted. If search pattern does contain matching groups only matching groups will be outputted (same line with no delimiter). [boolean] |
| -T | --trim-pipe Trim piped data before processing. If piped data only consists of chars that can be trimmed (new line, space, tabs...) it will become an empty string. [boolean] |
| | --trim-file Trim the content of the replacement file. [boolean] |
| | --trim Shorthand for --trim-pipe and --trim-file. [boolean] |
| | --js-pipe Treat the replacement as javascript, and give the code the piped data as pipe. [boolean] |
| -g | --glob-pipe Each line of the piped data is a path or a glob of the files to work on. They are used together with the ones given as arguments. [boolean] |
| | --null-in The piped paths and globs of --glob-pipe are separated by null bytes, like from find -print0 or --null-out. [boolean] |
| -R | --replacement-pipe Replacement will be piped in. You still need to provide a dummy value (like _) as replacement parameter. [boolean] |
| -j | --replacement-js, --js Treat replacement as javascript source code. The statement from the last expression will become the replacement string. Purposefully implemented the most insecure way possible to remove any incentive to consider running code from an untrusted part. The full match will be available as a javascript variable named $0 while each captured group will be available as $1, $2, $3, ... and so on. At some point, the $ char will give you a headache when used from the command line, so use €0, €1, €2, €3... instead. The code has access to the following variables: r as an alias for require with both expanded to understand a relative path even if it is not starting with ./, fs from node, path from node, globs from npm, pipe: the data piped into the command with --js-pipe (null if no piped data, and pipe_ is ❌ ), find: pattern searched for (the needle), text: full text being searched i.e. file content or piped data (the haystack), bytes: total size of the haystack in bytes, size: human-friendly representation of the total size of the haystack, time: String representing the local time when the command was invoked, time_obj: date object representing time, now: alias for time, cwd: current process working dir, nl: a new-line char, _: a single space char (for easy string concatenation). The following values defaults to ❌ if haystack does not originate from a file: file: contains the full path of the active file being searched (including full filename), file_rel: contains file relative to current process working dir, dirpath: contains the full path without filename of the active file being searched, dirpath_rel: contains dirpath relative to current process working dir, filename: is the full filename of the active file being searched without path, name: filename of the active file being searched with no extension, ext: extension of the filename including leading dot, mtime: ISO inspired representation of the last local modification time of the current file, ctime: ISO representation of the local creation time of the current file. mtime_obj: date object representing mtime, ctime_obj: date object representing ctime. The numbers: $n: the number of the file, $t: the number of files, $i: the number of the file among the files with a match so far, $c: among the files changed so far, $m: the number of the match in the file, $M: across all files. Each has cousins with zeros in front ($n0 has 2 digits, $n00 3, and so on to 10) and with a space at the end ($n_). For speed, the code runs once per file unless it names the match, a group, $m or $M. Access a named group via the name prepended with rr_. (?<year>...) gives you rr_year. A named group name can not end with _. All variables, except from module, date objects, nl and _, has a corresponding variable name followed by _ where the content has an extra space at the end (for easy concatenation). [boolean] |
| -x | --exclude-re Files with a path that matches this regular expression will be excluded from processing. Can be provided multiple times. [string] |
| -X | --exclude-glob Files found with this glob will be excluded from being processed. Can be provided multiple times and will act as one group of globs, meaning you can use ! notation to negate. [string] |
| -h | --help Display help. [boolean] |
Files identified for potential processing are identified by:
- Group A: Input values specifying existing files
- Group B: Input values that are not specifying an existing file are globbed as a pool (meaning you can use the
!notation to take files away from group B, but not from group A) - Group C: All
-Xvalues are globbed as a pool (meaning you can use the!notation) - Subtract group C from group A and B (the files are compared by their full path)
- Remove any file paths that matches any
-xprovided regex. Use--full-pathto look for the full path. - All globs are case-insensitive, like the search, unless
--void-ignore-caseis set. - The lines piped in with
--glob-pipeare input values too, just like the arguments. - The files are sorted with
--sort: natural by default, sofile2comes beforefile10, and case is ignored.
- Patterns are described as javascript notation regex
- Pattern defaults to global multiline case-insensitive search
- Supports regex lookaheads in the pattern
- Supports backreference to matching groups in the replacement
- Data can be piped in, both as the text to search and as the replacement
- The names of the changed files are printed to stdout, so they can be piped on (to stderr with
--outputand--output-match) - Set the delimiter between names and between the content of files
- Check the result in a CI step with
--expect-files,--expect-diff,--expect-zero-diffand--expect-diff-all. Add--simulateto only check - Rename files with a search and replace on their name (
--rename), or move them with one on their path (--rename-path). If planned renames clash the process is halted and no change is applied to any file, and--simulateshows each old name next to its new one - See the release note for a log of changes. Descriptions are given in latest patch version.
-
Options can only be set after the replacement parameter. "But I like to put my options as the first thing, so I know what I am doing" I agree, but we must sometimes sacrifice habits for consistency.
-
Per default
€is treated as an alias for$in the pattern, the replacement and--exclude-re. The main reason is for you not to worry about how command line tools often have a special relationship with the$char. You can escape your way out of this old love story, but it often pops up in unexpected ways. Use the-€flag if you need to search or replace the actual euro char. -
Per default
§is treated as an alias for\in the pattern, the replacement and--exclude-re. The main reason is for you not to worry about escaping the escape of an escape. Use the-§flag if you need to search or replace the actual section char. -
The
€and§aliases apply to pattern, replacement,--exclude-revalues and delimiters. File globs,--exclude-glob, the path of--replacement-fileand piped data are used exactly as they are, so a file namedprice€.txtis found without any flags. -
The default delimiter is a magic newline ensuring that there is at least one
\nafter printed file content, matches or names. This means that if the content already ends on a\nno additional\nis added. This behaviour can not be recreated with other delimiter values. -
Piped data that is printed is left as it came in, so no delimiter is added after the end of it. Matches from
--output-matchare different: they are new output, so they get the delimiter whether they come from a file or from piped data. -
Finding no files is a warning (🟡) and not an error, so the exit code is 0. Nothing went wrong, there was just nothing to do.
| Code | Meaning |
|---|---|
| 0 | Done. Warnings (🟡) are ok. |
| 1 | A check from --expect-* was not met. |
| 2 | One or more files gave an error (❌), and the run went on. |
| 3 | The run stopped (❌ … No further processing.): a bad pattern, an unknown encoding, wrong flags, planned renames that clash, or any error with --halt. |
Like grep and diff, code 1 means that RexReplace did its job and the answer is no, and 2 or 3 means that something went wrong.
- RexReplace reads each file fully into memory, so working on your 8Gb log files will probably not be ideal.
- Files are read as utf8 by default. Null bytes are kept as they are but for binary data in general not all bytes are valid and becomes
�if changed. Use--encoding latin1to run binary safe: every byte is kept, but note that because that means one byte per char search and replace only works with ASCII chars (no utf8 or emojis). - For versions of Node prior to 6, please use version 2.2.x. For versions of Node prior to 0.12, please use the legacy version of RexReplace called rreplace
- Flexibility regarding text pattern matching
- Easy to filter what files to be treated
- Helpful interface
- Tests (if you know how to do a test cover report on javascript code ran via the command line, please let me know)
- Speed. Well... obviously, speed is important, but to what extent does a 29 millisecond command really satisfy the user compared to a 294 millisecond command? See test->speed for more info.
> time cat README.md | sed 's/a/x/g' > /dev/null
cat myfile 0,00s user 0,00s system 45% cpu 0,011 total
sed 's/a/x/g' > /dev/null 0,00s user 0,00s system 43% cpu 0,029 total
> time rr a x README.md -o > /dev/null
rr x y myfile -o > /dev/null 0,21s user 0,04s system 86% cpu 0,294 totalEach part of a run is a stack of layers. Each layer is optional, and works on what the layer before gives it. So the flags work together by their order, not by special rules.
- The pattern as typed.
€is$and§is\. --literal(-L) makes it plain text, so.is a dot.--void-ignore-case(-I),--void-multiline(-M),--dot-all(-s),--unicode(-u) and--void-global(-G) decide how the regex works.
> echo -n 'a.b' | rexreplace '.' '_'
___
> echo -n 'a.b' | rexreplace '.' '_' --literal
a_b- The replacement as typed:
€is$,§is\, and escapes like\nand\tare read. In--jscode javascript reads the escapes, and the path of--replacement-fileis used as it is, with no aliases or escapes. Or the piped data with--replacement-pipe(-R), used as it is. -
--js(-j) runs it as code, and the value of the code goes on.--js-pipealso gives the code the piped data aspipe. -
--replacement-file: what came so far is the path of a file, and the content of the file goes on. -
--trim-filetrims it. - It goes into the text.
$1is group 1,$&the whole match, and$$one$, unless the value came from--jscode, or--literal-replacement(-l) is given.
> echo -n foo | rexreplace '(f)(oo)' '$2$1'
oof
> echo -n foo | rexreplace 'o' '$0.toUpperCase()' --js
fOO
> echo -n PRICE | rexreplace '(PRICE)' 'only $1.99'
only PRICE.99
> echo -n PRICE | rexreplace '(PRICE)' 'only $1.99' --literal-replacement
only $1.99
> echo -n 'a {{footer}} b' | rexreplace '\{\{footer\}\}' partials/footer.html --replacement-file --trim-file
a <footer> bAs the layers are in order, they can be put together. Here the piped data is code that gives the path of the file to use:
> echo '"partials/footer.html"' | rexreplace '\{\{footer\}\}' _ page.html -R --js --replacement-file --trim-fileCode that lives in a file is loaded with r() in --js, like 'r("fix")($0)'.
- The paths and globs given as arguments, and the lines piped in with
--glob-pipe(-g).--null-insplits the piped lines on null bytes. - A path that is a file is used as it is. The rest are globs.
- The globs ignore case, unless
--void-ignore-case(-I). --exclude-glob(-X) takes away the files its globs find.--full-path(-f) makes each path full.--exclude-re(-x) takes away the files with a path that matches.--sortdecides the order, and--sort-reverseturns it around.
> rexreplace foo bar 'docs/**/*.md' --exclude-glob 'docs/old/**' --exclude-re draft --sort size- What is printed, and where:
| Flags | stdout | stderr |
|---|---|---|
| (none) | the names of the changed files | |
--output (-o) |
the content | the names |
--output-match (-m) |
the matches | the names |
--rename --simulate |
the new names | the old names |
> rexreplace o x a.file b.file -o 2>/dev/null
fxx
fxx
> rexreplace o x a.file b.file -o 2>&1 >/dev/null
a.file
b.file- Between the items:
--name-delimitergoes after each name, and--content-delimiterafter each content or match.--delimitersets both, and--null-outis--delimiterwith a null byte. The most specific one wins. Without a delimiter it is a line break, unless the item already ends with one. - The tails: the delimiter also goes after the last item, unless
--no-name-delimiter-tail,--no-content-delimiter-tailor--no-tail. Piped data that is printed is left as it came in, with nothing after it.
> rexreplace o x a.file b.file --name-delimiter ', ' --no-name-delimiter-tail --sort natural
a.file, b.file
> rexreplace o x a.file b.file --null-out | xargs -0 ls -l--nlends stdout and stderr with a line break, if they do not end with one.- How much:
--quiet(-q) hides the names and the warnings (🟡), but not the errors (❌), and not the content of--outputor the matches of--output-match.--quiet-total(-Q) shows nothing, and--verbose(-V) tells each step to stderr. An error that stops the run ends withNo further processing. - The status report:
--statusand--status-compact(-c) print it as the last thing, also with--quiet-total.--debug(-d) prints the status report, with what went into the engine and what it did, and hides the rest, except the content of--outputand the matches of--output-match. - The exit code, see Exit codes.
-
--rename(-r) works on the name of each file, and--rename-pathon the whole path, so a/moves the file. If planned renames clash the process is halted and no change is applied to any file. -
--output-matchonly prints what the pattern finds. The replacement is not used, so--jscode is not run, a replacement file is not read, and--renamerenames nothing. - The files are handled at the same time, so names can come in another order than the sort. They are handled one at a time, in order, with
--output,--output-match,--void-async(-A),--sort, or when--jscode uses$i,$c,$mor$M.
The tests come in five groups:
- The demo in test/cli/run.sh shows how RexReplace is used, with a few examples of each feature. It is run with the installed commands:
rexreplacewith Node andrexreplace-bunwith Bun, by the scripts in test/runtime. You must have both. - CLI tests in
test/cli/*.test.jsrun the CLI with Bun's shell, and check stdout, stderr and the exit code. This is where combinations of flags are checked. - Engine tests in
test/engine/*.test.jscall the engine directly (no build and no CLI involved), so they are fast enough to run while developing. There is one file per flag, named after the long version of the flag. - Fix tests in
test/fix/<issue>.test.jstest that an issue stays fixed, with the engine, the CLI or both. - Fuzzy tests in
test/fuzzy/*.test.jsfeed the engine random input and compare the result with what plain javascript gives.
The build and the CLI, engine, fix and fuzzy tests are run by bun. After git clone'ing the repo and yarn install'ing you can invoke the tests with:
> yarn test # build + demo + CLI, engine and fix tests + fuzzy tests
> yarn test-js # only the CLI, engine and fix tests
> yarn test-fuzzy # only the fuzzy tests
> yarn test-cli # only the demo (build first)The fuzzy tests get new input on every run, so they are not part of what the CI pipeline runs (yarn test-ci). The seed is printed, so a failing run can be repeated:
> RR_FUZZ_SEED=1234 yarn test-fuzzytl;dr:
Files over 5 Mb are faster with rr than with sed - but - it does not matter as any file under 25 Mb has less than 0.7 seconds in difference.
The speed test is initiated by npm run test-speed. The test takes files in different sizes and compares the processing time for RexReplace (rr) and the Unix tool sed. The test uses the sources of a website displaying the book 1984 by George Orwell. The task for the tests is to remove all HTML tags by search-and-replace, so only the final text is left. The source is 888Kb, so all files up to 500Kb are generated directly from the source, while larger files are created by combining the first 500Kb several times. Each test runs 10 times to even out any temporary workload fluctuations.
Results from the latest speed test can always be seen in the speed test log.
The graph visualises speed as relative to fastest overall run (sed on a 1kb file). This chart also has an interactive version in log scale, so the details in the low end can be studied better. Interestingly files of 1Kb and 5Kb take longer for rr than 10Kb files.
Now, what is relevant to notice is how sed only takes 4.9 seconds longer for the 100Mb file - even if the difference looks drastic on the graph.
Speed relative to fastest tool for each file size
---------------------------------------------------
Bytes sed rr Diff (seconds)
1 1 39 0,5 <= sed is 39x faster
5 1 32 0,4
10 1 27 0,4
100 1 19 0,3
500 1 7 0,3
1000 1 4 0,3
5000 1 1 0,0 <= same speed for 5Mb file
10000 2 1 0,3
25000 2 1 1,1
50000 3 1 3,1
100000 3 1 4,9 <= rr 3.1.0 is 3x faster
So even though the speed evolves very differently, there is only little practical use of the focus on speed for most use cases. Replacing in 10.000 small files? Use RexReplace and get yourself a cup of coffee - or spend half an hour getting sed to work as you want it to and enjoy the thrilling few seconds it takes to do its magic.
Please note that speeds might look very different when files get as large as the memory available.
Reading many files multiple times is far more time-consuming than creating a complex regex and reading each file once.
Anecdote time
Imagine you need to duplicate a set of files, but within the content are references (let's call them keys) that must be unique across both old and new files. We have the complete list of keys so a reasonable approach is to append
_v2to each old key in the new files by running a separaterexreplacecommand for each key.However, in a real-world scenario with 5,000 keys across 10,000 files, this approach took 57 minutes. The bottleneck was identifying the 10,000 files 5,000 times, open + read each file 5000 times and the startup time of node x 5,000.
Seeking to optimize performance we bundled all keys into a single regex search pattern like
(key1|key2|key3|...)'. This resulted in a nasty 300KB one-line command but it reduced the runtime to 2.14 seconds!# Separate commands for each key rexreplace key1 key1_v2 src/v2/**/*.* rexreplace key2 key2_v2 src/v2/**/*.* rexreplace key3 key3_v2 src/v2/**/*.* ... rexreplace key5000 key5000_v2 src/v2/**/*.* # Total time: 57 minutes # Single command for all keys rexreplace '(key1|key2|key3|...|key5000)' '$1_v2' src/v2/**/*.* # Total time: 2.14 secondsLook, not all problems can be solved like this, but the moral is still valid: Choosing the right approach can save you a significant amount of time.
.oO(What should "sed" have looked like by now?)
- Default to RegExmoji lax syntax
- Support RegExmoji strict syntax
- Add support to require hjson, jsonh, json5, yaml, ini files directly
- Test-run with info outputted about what will happen (sets -t and does not change anything)
- Let search and replace be within the names of the files (ask for overwriting. -Y = no questions)
- Let search and replace be within the path of the files (ask for overwriting. -Y = no questions)
- Let pattern and globs be piped
- Let pipe be placed into search or replace.
- Let Pattern, replacement, and globs come from a file
- Let pattern and glob be javascript code returning a string as the result
- Flag for String-literal (no regex, no special chars, no escape chars) to avoid backslashes or remembering which characters needs to be escaped
- Check if https://github.com/eugeneware/replacestream is good to rely on
- Check if regex engine from spider monkey can be wrapped in something that does not need node
- Check if sd can be wrapped in a WASM module
- Set engine to RE2 https://www.npmjs.com/package/re2
- Set engine to run on streams with https://www.npmjs.com/package/replacestream
- Implement in go so that all platforms can be supported with no need for node (might be based on)
- Let https://github.com/dthree/vorpal deal with the interface? Or maybe https://www.npmjs.com/package/pretty-cli
- Load env. vars from prefix setting before loading data. Check conflict also.
Many projects are seeking to solve the same problem as RexReplace. In my opinion they all lack the flexibility of the RexReplace CLI interface and diverse the replacement can be, but have other strong features. If our way does not suit you, we suggest you have a look at:
- sd - Blazingly fast! 5.9x faster than RexReplace. Almost perfect. Using "Good ol' unix philosophy to the rescue." for pinpointing files so not always easy. Get it via cargo for Rust.
- replace-in-file - same speed as RexReplace. Very good support to be used as a module for a node application. A bit cumbersome CLI notation for pinpointing files. Get via npm.
- replace-x - 3.8x slower than RexReplace. A bit cumbersome CLI notation for pinpointing files. Get it via npm.
- replace - 2.6x slower than RexReplace. Very chatty default output. A bit more cumbersome CLI notation for pinpointing files. Get it via npm.
Please note that RexReplace is an OPEN open source software project. This means that individuals making significant and valuable contributions are given commit access to the project to contribute as they see fit. This project is more like an open wiki than a standard guarded open source project.
Icon inspired by Freepik from www.flaticon.com
