入行运维很重要的一步就是学习Linux系统。正所谓会了不难，难了不会，很多初学者都因Linux命令和Shell […]

1.脚本自杀正文有些时候我们写的shell脚本中有一些后台任务，当脚本的流程已经执行到结尾处或将其kill掉时

PowerShell is an interactive shell and scripting language from Microsoft. It’s object-oriented — and that’s not just a buzzword, that’s a big difference to how the standard Unix shells work. And it is actually usable as an interactive shell. Getting Started PowerShell is so nice, Microsoft made it twice. Specifically, there concurrently exist two products named PowerShell: Windows PowerShell (5.1) is a built-in component of Windows. It is proprietary, Windows-only, and is based on the equally proprietary and equally Windows-only .NET Framework 4.x. It has a blue icon. PowerShell (7.x), formerly known as PowerShell Core, is a stand-alone application. It is MIT-licensed (developed on GitHub), available for Windows, Linux, and macOS, and is based on the equally MIT-licensed and equally multi-platform .NET (formerly .NET Core). It has a black icon. Windows PowerShell development stopped when PowerShell (Core) came out. There are some niceties and commands missing in it, but it is still a fine option for trying it out or for when one can’t install PowerShell on a Windows system but need to solve something with code. All examples in this post should work in either version of PowerShell on any OS (unless explicitly noted otherwise). Install the modern PowerShell: Windows, Linux, macOS. Objects? In my shell? Let’s try getting a directory listing. This is Microsoft land, so let’s try the DOS command for a directory listing — that would be dir: PS C:\tmp\hello> dir Directory: C:\tmp\hello Mode LastWriteTime Length Name ---- ------------- ------ ---- d---- 2024-04-29 18:00 world -a--- 2024-04-29 18:00 23 example.py -a--- 2024-04-29 18:00 7 foobar.txt -a--- 2024-04-29 18:00 14 helloworld.txt -a--- 2024-04-29 18:00 0 newfile.txt -a--- 2024-04-29 18:00 5 test.txt This looks like a typical (if slightly verbose) file listing. Now, let’s try to do something useful with this. Let’s get the total size of all .txt files. In a Unix shell, one option is du -bc *.txt. The arguments: -b (--bytes) gives the real byte size, and -c (--summarize) produces a total. The result is this: 7 foobar.txt 14 helloworld.txt 0 newfile.txt 5 test.txt 26 total But how to get just the number? This requires text manipulation (getting the first word of the last line). Something like du -bc *.txt | tail -n 1 | cut -f 1 will do. There’s also wc --total=only --bytes *.txt — but this is specific to GNU wc, so it won’t cut it on *BSD or macOS. Another option would be to parse the output of ls -l — but that might not always be easy, and the output may contain something unexpected added by the specific ls version or the user’s specific shell configuration. Let’s try something in PowerShell. If we do $x = dir, we’ll have the output of the dir command in $x. Let’s try to analyse it further, is the first character a newline? PS C:\tmp\hello> $x = dir PS C:\tmp\hello> $x[0] Directory: C:\tmp\hello Mode LastWriteTime Length Name ---- ------------- ------ ---- d---- 2024-04-29 18:00 world That’s interesting, we didn’t get the first character or the first line, we got the first file. And if we try $x[1]? PS C:\tmp\hello> $x[1] Directory: C:\tmp\hello Mode LastWriteTime Length Name ---- ------------- ------ ---- -a--- 2024-04-29 18:00 23 example.py What if we try getting the Length property out of that? PS C:\tmp\hello> $x[1].Length 23 It turns out that dir returns an array of objects, and PowerShell knows how to format this array (and a single item from the array) into a nice table. What can we do with it? This: PS C:\tmp\hello> Get-ChildItem -Filter '*.txt' | ForEach-Object { $_.Length } | Measure-Object -Sum Count : 4 Average : Sum : 26 Maximum : Minimum : StandardDeviation : Property : PS C:\tmp\hello> (Get-ChildItem -Filter '*.txt' | ForEach-Object { $_.Length } | Measure-Object -Sum).Sum 26 PS C:\tmp\hello> (Get-ChildItem -Filter '*.txt' | Measure-Object -Sum -Property Length).Sum 26 PS C:\tmp\hello> (Get-ChildItem -Recurse -Filter '*.txt' | Measure-Object -Sum -Property Length).Sum 30 PS C:\tmp\hello> $measured = (Get-ChildItem -Recurse -Filter '*.txt' | Measure-Object -Sum -Property Length) PS C:\tmp\hello> $measured.Sum / $measured.Count 6 We can iterate over all file objects, get their length (using ForEach-Object and a lambda), and then use Measure-Object to compute the sum (Measure-Object returns an object, we need to get its Sum property). We can replace the ForEach-Object call with the -Property argument in Measure-Object. And if we want to look into subdirectories, we can easily add -Recurse to Get-ChildItem. We get actual integers we can do math on. You might have noticed I used Get-ChildItem instead of dir in the previous example. Get-ChildItem is the full name of the command (cmdlet). dir is one of its aliases, alongside gci and ls (Windows-only to avoid shadowing /bin/ls). Many common commands have aliases defined for easier typing and ease of use — Copy-Item can be written as cp (for compatibility with Unix), copy (for compatibility with MS-DOS), and ci. In our examples, we could also use measure for Measure-Object and foreach or % for ForEach-Object. Those aliases are a nice thing to have for interactive use, but for scripts, it’s best to use the full names for readability, and to avoid depending on the environment for those aliases. More filesystem operations Files per folder There’s a photo collection in a Photos folder, grouped into folders. The objective is to see how many .jpg files are in each folder. Here’s the PowerShell solution: PS C:\tmp> Get-ChildItem Photos/*/*.jpg | Group-Object { $_.Directory.Name } | Sort-Object -Property Count -Descending Count Name Group ----- ---- ----- 10 foo bar {C:\tmp\Photos\foo bar\img001.jpg, C:\tmp\Photos\foo bar\img002.jpg, C:\tmp\Photos\foo bar\img003.jpg…} 2 example {C:\tmp\Photos\example\img101.jpg, C:\tmp\Photos\example\img201.jpg} In Unix land, StackOverflow has a lot of solutions. The top solution is du -a | cut -d/ -f2 | sort | uniq -c | sort -nr — a lot of tools mashed together, starting with a tool to check disk usage, and a lot of string manipulation. The second solution uses find, read, and shell globbing. The PowerShell solution is quite simple and obvious to anyone who has ever touched SQL. The above example works for one level of nesting. For more levels, given Photos\one\two\three.jpg, use Get-ChildItem -Filter '*.jpg' -Recurse Photos, and: Group by $_.Directory.Name (same as before) to get two Group by Split-Path -Parent ([System.IO.Path]::GetRelativePath("$PWD/Photos", $_.FullName)) to get one/two Group by ([System.IO.Path]::GetRelativePath("$PWD/Photos", $_.FullName)).Split([System.IO.Path]::DirectorySeparatorChar)[0] to get one (All of the above examples work for a single folder as well. The latter two examples don’t work on Windows PowerShell.) Duplicate finder Let’s build a simple tool to detect byte-for-byte duplicated files. Get-FileHash is a shell built-in. We can use Group-Object again, and Where-Object to filter only matching objects. Computing the hash of every file is quite inefficient, so we’ll group by the file length first, and then ensure the hashes match. This gives us a nice pipeline of 6 commands: # Fully spelled out Get-ChildItem -Recurse -File | Group-Object { $_.Length } | Where-Object { $_.Count -gt 1 } | ForEach-Object { $_.Group } | Group-Object { (Get-FileHash -Algorithm MD5 $_).Hash } | Where-Object { $_.Count -gt 1 } # Using aliases gci -Recurse -File | group { $_.Length } | where { $_.Count -gt 1 } | foreach { $_.Group } | group { (Get-FileHash -Algorithm MD5 $_).Hash } | where { $_.Count -gt 1 } # Using less readable aliases gci -Recurse -File | group { $_.Length } | ? { $_.Count -gt 1 } | % { $_.Group } | group { (Get-FileHash -Algorithm MD5 $_).Hash } | ? { $_.Count -gt 1 } Serious Scripting: Software Bill of Materials Software Bills of Materials (SBOMs) and supply chain security are all the rage these days. The boss wants to have something like that, i.e. a CSV file with a list of packages and versions, and only the direct production dependencies. Sure, there exist standards like SPDX, but the boss does not like those pesky “standards”. The backend is written in C#, and the frontend is written in Node.js. Since we care only about the production dependencies, we can look at the .csproj and package.json files. For Node packages, we’ll also try to fetch the license name from the npm API (the API is a bit more complicated for NuGet, so we’ll keep it as a TODO in this example). $ErrorActionPreference = "Stop" # stop execution on any error Set-StrictMode -Version 3.0 function Get-CsprojPackages([string]$Path) { return Select-Xml -Path $Path -XPath '//PackageReference' | ForEach-Object { [PSCustomObject]@{ Name = $_.Node.GetAttribute("Include") Version = $_.Node.GetAttribute("Version") Source = 'nuget' License = 'TODO' } } } function Get-NodePackages([string]$Path) { $nameToVersion = (Get-Content -Raw $Path | ConvertFrom-Json).dependencies return $nameToVersion.psobject.Properties | ForEach-Object { [PSCustomObject]@{ Name = $_.Name Version = $_.Value Source = 'node' License = (Get-NodeLicense -Name $_.Name) } } } function Get-NodeLicense([string]$Name) { try { return (Invoke-RestMethod -TimeoutSec 3 "https://registry.npmjs.org/$Name").license } catch { return "???" } } $csprojData = @(Get-ChildItem -Recurse -Filter '*.csproj' | ForEach-Object { Get-CsprojPackages $_.FullName }) $nodeData = @(Get-ChildItem -Recurse -Filter 'package.json' | Where-Object { $_.FullName -notlike '*node_modules*' } | ForEach-Object { Get-NodePackages $_.FullName }) $allData = $csProjData + $nodeData $allData | ConvertTo-Csv -NoTypeInformation | Tee-Object sbom.csv Just like every well-written shell script starts with set -euo pipefail, every PowerShell script should start with $ErrorActionPreference = "Stop" so that execution is stopped as soon as something goes wrong. Note that this does not affect native commands, you still need to check $LASTEXITCODE. Another useful early command is Set-StrictMode -Version 3.0 to catch undefined variables. For .csproj files, which are XML, we look for PackageReference elements using XPath, and then build a PSCustomObject out of a hashmap — extracting the appropriate attributes from the PackageReference nodes. For package.json, we read the file, parse the JSON, and extract the properties of the dependencies object (it’s a map of package names to versions). To get the license, we use Invoke-RestMethod, which takes care of parsing JSON for us. In the main body of the script, we look for the appropriate files (skipping things under node_modules) and call our parser functions. After retrieving all data, we concatenate the two arrays, convert to CSV, and use Tee-Object to output to a file and to standard output. We get this: "Name","Version","Source","License" "AWSSDK.S3","3.7.307.24","nuget","TODO" "Microsoft.AspNetCore.SpaProxy","7.0.17","nuget","TODO" "@testing-library/jest-dom","^5.17.0","node","MIT" "@testing-library/react","^13.4.0","node","MIT" "@testing-library/user-event","^13.5.0","node","MIT" "@types/jest","^27.5.2","node","MIT" "@types/node","^16.18.96","node","MIT" "@types/react","^18.3.1","node","MIT" "@types/react-dom","^18.3.0","node","MIT" "react","^18.3.1","node","MIT" "react-dom","^18.3.1","node","MIT" "react-scripts","5.0.1","node","MIT" "typescript","^4.9.5","node","Apache-2.0" "web-vitals","^2.1.4","node","Apache-2.0" Could it be done in a different language? Certainly, but PowerShell is really easy to integrate with CI, e.g. GitHub Actions or Azure Pipelines. On Linux, you might be tempted to use Python — and you could get something done equally simply, as long as you don’t mind using the ugly urllib.request library, or alternatively ensuring requests is installed (and then you get into the hell that is Python package management). Using .NET classes PowerShell is built on top of .NET. This isn’t just the implementation technology — PowerShell gives access to everything the .NET standard library offers. For example, the alternate ways to group photos in multiple subdirectories we’ve explored above involve a call to a static method of the .NET System.IO.Path class. Other .NET types are also available. Need a HashSet? Here goes: PS> $set = New-Object System.Collections.Generic.HashSet[string] PS> $set.Add("hello") True PS> $set.Add("hello") False PS> $set.Add("world") | Out-Null PS> $set.Count 2 PS> $set -contains "hello" True PS> $set -contains "world" False It is also possible to load any .NET DLL into PowerShell (as long as it’s compatible with the .NET version PowerShell is built against) and use it as usual from C# (although possibly with slightly ugly syntax). Sick Windows Tricks Microsoft supposedly killed off Internet Explorer last year. Attempting to launch iexplore.exe will bring up Microsoft Edge. But you see, Internet Explorer is a crucial part of Windows, and has been so for over two decades. Software vendors have built software that depends on IE being there and being able to show web content. Some of them are using web views, but some of them prefer something else: COM. COM, or Component Object Model, is Microsoft’s thing for interoperability between different applications and/or components. COM is basically a way for classes offered by different vendors and potentially written in different languages to talk to one another. Under the hood, COM is C++ vtables plus standard reference counting and class loading/discovery mechanisms. The .NET Framework, and its successor .NET, have always included COM interoperability. The modern WinRT platform is COM on steroids. Coming back to Internet Explorer, it exposes some COM classes. They were not removed with iexplore.exe. This means you can bring up a regular Internet Explorer window in just two lines of PowerShell: $ie = New-Object -ComObject InternetExplorer.Application $ie.Visible = $true Why would you do that? The InternetExplorer.Application object lets you control the browser, e.g. you can use $ie.Navigate("https://example.com/") to go to a page. Why would you want to launch IE in 2024? I don’t know, I guess you can use it to laugh in the faces of the Microsoft developers who removed the user-accessible shortcuts? But there definitely exist some legacy applications that expect a COM-controllable IE. We have already explored the possibility of using classes from .NET. .NET comes with a GUI framework named Windows Forms, which can be loaded from PowerShell and used to build a GUI. There is no form designer, so it requires manually defining and positioning controls, but it actually works. PowerShell can also do various Windows management tasks. It can manage boot settings, BitLocker, Hyper-V, networking, storage… For example, to get the percentage of disk space remaining: $c = Get-Volume C "$(($c.SizeRemaining / $c.Size) * 100)%" Getting out of PowerShell land As a shell, PowerShell can obviously launch subprocesses. Unlike something like Python, running a subprocess is as simple as running anything else. If you need to git pull, you just type that. Or you can make PowerShell interact with non-PowerShell commands, reading output and passing arguments: $changes = (git status --porcelain --null) if ($LASTEXITCODE -eq 128) { throw "Not a git repository" } elseif ($LASTEXITCODE -ne 0) { throw "Getting changes from git failed" } if ($null -eq $changes) { Write-Host "No changes found" } else { $untrackedFiles = @( $changes.Split("`0") | Where-Object { $_.StartsWith('?? ') } | ForEach-Object { $_.Remove(0, 3) } ) # Alternate spelling for regex fans: $untrackedFilesForRegexFans = @( $changes.Split("`0") | Where-Object { $_ -match '^\?\? ' } | ForEach-Object { $_ -replace '^\?\? ','' } ) if ($untrackedFiles) { Write-Host "Opening $($untrackedFiles.Length) untracked files in VS Code" code $untrackedFiles } else { Write-Host "No untracked files" } } I chose to compute untracked files with the help of standard .NET string manipulation methods, but there’s also a regex option. On a related note, there are three content check operators: -match uses regex, -like uses wildcards, and -contains checks collection membership. Profile script I use a fairly small profile script that adds some behaviours I’m used to from Unix, and to make Tab completion show a menu. Here are the most basic bits: Set-PSReadLineOption -HistorySearchCursorMovesToEnd Set-PSReadLineKeyHandler -Key UpArrow -Function HistorySearchBackward Set-PSReadLineKeyHandler -Key DownArrow -Function HistorySearchForward Set-PSReadlineKeyHandler -Key ctrl+d -Function DeleteCharOrExit Set-PSReadlineKeyHandler -Key Tab -Function MenuComplete Set-PSReadLineOption -AddToHistoryHandler { param($command) # Commands starting with space are not remembered. return -not ($command -like ' *') } Apart from that, I use a few aliases and a pretty prompt with the help of oh-my-posh. The unusual and sometimes confusing parts PowerShell can be verbose. Some of its syntax is a little quirky, compared to other languages, e.g. the equality and logic operators (for example, -eq, -le, -and). The aliases usually help with remembering commands, but they can’t always be depended on — ls is defined as an alias only on Windows, and Windows PowerShell aliases wget and curl to Invoke-WebRequest, even though all three have completely different command line arguments and outputs (this was removed in PowerShell). Moreover, the Unix/DOS aliases do not change the argument handling. rm -rf foo is invalid. rm -r foo is, since argument names can be abbreviated as long as the abbreviation is unambiguous. rm -r -f foo is not valid, because -f can be an abbreviation of -Filter or -Force (so rm -r -fo foo) will do. rm foo bar does not work, an array is needed: rm foo,bar. C:\Windows\regedit.exe launches the Registry editor. "C:\Program Files\Mozilla Firefox\firefox.exe" is a string. Launching something with spaces in its name requires the call operator: & "C:\Program Files\Mozilla Firefox\firefox.exe". PowerShell’s tab completion will add the & if necessary. There are two function call syntaxes. Calling a function/cmdlet uses the shell-style syntax with argument names: Some-Function -Arg1 value1 -Arg2 value2, and argument names can be abbreviated, and can sometimes be omitted. Calling a method requires a more traditional syntax: $obj.SomeMethod(value1, value2). Names are case-insensitive in either case. The escape character is the backtick. The backslash is the path separator in Windows, so making it an escape character would make everything painful on Windows. At least it makes it easy to write regex. The ugliest part The ugliest and the least intuitive part of PowerShell is the handling of single-element arrays. PowerShell really wants to unpack them to a scalar. The command (Get-ChildItem).Length will produce the number of files in the current directory — unless there is exactly one file, in which case it will produce the single file’s size in bytes. And if there are zero items, instead of an empty array, PowerShell produces $null. Sometimes, things will work out in the end (since many cmdlets are happy to get either as inputs), but sometimes, PowerShell must be asked to stop this madness and return an array: @(Get-ChildItem).Length. The previous example with git status leverages its --null argument to get zero-delimited data, so we expect either $null or a single string according to the rules. If we didn’t want to use --null, we would need to use @(git status --porcelain) to always get an array (but we would also need to remove quotes that git adds to paths that contain spaces). Conclusion PowerShell is a fine interactive shell and scripting language. While it does have some warts, it is more powerful than your usual Unix shell, and its strongly-typed, object-oriented code beats stringly-typed sh spaghetti any day.

上次出了一篇linux 命令详解，得到了很多小伙伴的认可，有部分粉丝私信我，让我出一份shell 编程。进过一段时间准备，花了2周时间，整理了一篇shell 入门到实战的博文，欢迎大家阅读，指点。

Linux 的发行版有很多，这里罗列7个漂亮的 Linux 发行版，可以说是Linux操作系统界的颜值担当了。

虽然网络上有很多对 Shell 的正式定义，但如果你已经熟悉 Linux 命令，编写 Shell 脚本并不算难事。Shell 本质上就是 Linux 命令的集合，将一系列命令组合在一起，以达成特定的目标，这便构成了 Shell 脚本。通过这种方式，工作的负担得以减轻，工作效率也得以提升。

在 shell 脚本中成功应用 sed 编辑器和 gawk 程序的关键在于熟练掌握正则表达式。正则表达式并非易事，从海量数据中提取出特定信息往往是一项复杂的任务，因此可能需要耗费一些功夫。

On behalf of the team and everyone who has contributed, I'm happy to announce that Spring Shell 3.1.11, 3.2.4 and 3.3.0-M1 has been released and are now available from Maven Central and Milestone Repo respectively. Please see the release notes 3.1.11, release notes 3.2.4 and release notes 3.3.0-M1 for more details. Thanks to all those who have contributed with issue reports and pull requests. How can you help? Project Page | GitHub | Issues | Documentation

Linux/[Unix]系统提供了多种Shell，为何会如此多样化呢？

文本处理在Shell脚本中扮演着至关重要的角色，它使你能够从文本数据中提取出有价值的信息，并执行各种操作，如搜索、替换、格式化和过滤等。