Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,3 +52,26 @@ You can obtain `longhornctl` either through downloading a prebuilt binary or by
> - `longhornctl`: A command-line interface for remote Longhorn operations, designed to be run outside the Kubernetes cluster. It executes `longhornctl-local` for operations within the cluster.
> - `longhornctl-local`: A command-line interface to be used within a DaemonSet pod inside the Kubernetes cluster, handling in-cluster and host operations.

### V2-only prerequisites

The default `--data-engine-mode=v1-capable` retains iSCSI requirements. An explicitly selected `v2-only` mode omits iSCSI packages, modules and service operations, and requires SPDK checks and setup even without `--enable-spdk`.

Use matching CLI binaries and a CLI image containing this capability. The remote command passes an explicit flag to the local executable so an older image fails instead of silently running V1 checks.

```bash
longhornctl install preflight --data-engine-mode=v2-only --huge-page-size=2048 --allow-pci=none
longhornctl check preflight --data-engine-mode=v2-only --huge-page-size=2048
```

Installation success is prerequisite setup, followed by a separate host qualification check. A required reboot remains unfinished installation; reboot and rerun installation before checking. Restart kubelet when necessary for its Node capacity to reflect the hugepage allocation. V2 check failures return a nonzero exit status after printing the complete report.

V2 installation supports the known package-manager distributions. COS, Talos, unknown operating systems, unqualified NUMA layouts and host-provided SPDK startup hooks are rejected; configure immutable hosts through their own declarative prerequisites. The supported setup profile uses 2MiB pages on one existing NUMA node (node 0 by default). `HUGENODE` accepts one nonnegative integer. The requested memory, page size, PCI selection and driver must agree with any supplied `--spdk-options`; unknown, duplicate or conflicting options are rejected. Setup uses a bounded, cleared environment and verifies the actual allocation afterward.

An empty PCI selection is fenced as `none`. Explicit device selection requires unique full PCI addresses; do not select devices still in use. Installer `--driver-override` accepts `vfio-pci` or `uio_pci_generic`; checker `--userspace-driver` uses the kernel module names `vfio_pci` or `uio_pci_generic`.

V2 remote runs use separate resource names and collect complete reports from the identified DaemonSet and completed container instances. Cleanup deletes only objects created by that invocation, with UID preconditions. This mode neither converts existing volumes nor enables the manager's separate V2-only installation capability. Retire V1 data before removing its host prerequisites.

Remote V2 installation records its invocation under `/var/lib/longhorn-cli/preflight-attempts` before changing host prerequisites. A Kubernetes retry or replacement Pod reports an error instead of repeating that invocation's host writes. The small attempt marker remains for audit and replay protection; assess any partial installation or required reboot before explicitly starting a new command. Failed and reboot-pending attempts are transported as error reports and the remote command exits nonzero.

Replay protection requires a private installer-owned directory on a verified ext, XFS, Btrfs or F2FS filesystem. Volatile and unverified stores are refused. Directory entries and the marker are synced before host changes start.

15 changes: 14 additions & 1 deletion cmd/local/subcmd/check.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,10 @@ import (
"github.com/spf13/cobra"

"github.com/longhorn/cli/pkg/consts"
local "github.com/longhorn/cli/pkg/local/preflight"
"github.com/longhorn/cli/pkg/types"
"github.com/longhorn/cli/pkg/utils"

local "github.com/longhorn/cli/pkg/local/preflight"
)

func NewCmdCheck(globalOpts *types.GlobalCmdOptions) *cobra.Command {
Expand All @@ -35,14 +36,22 @@ func newCmdCheckPreflight(globalOpts *types.GlobalCmdOptions) *cobra.Command {
Long: `This command verifies your Kubernetes cluster environment to ensure it meets Longhorn's requirements. It performs a series of checks that can help identify potential issues that may prevent Longhorn from functioning correctly.`,

PreRun: func(cmd *cobra.Command, args []string) {
if err := validateLocalDataEngineEnvironment(cmd, localChecker.DataEngineMode); err != nil {
utils.CheckErr(err)
}
localChecker.LogLevel = globalOpts.LogLevel
localChecker.CollectionOnly = os.Getenv(consts.EnvPreflightCollectionOnly) == "true"

if err := localChecker.Init(); err != nil {
utils.CheckErr(errors.Wrap(err, "Failed to initialize preflight checker"))
}
},

Run: func(cmd *cobra.Command, args []string) {
if localChecker.DataEngineMode == string(types.DataEngineModeV2Only) {
utils.CheckErr(runAndOutputV2Preflight(&localChecker))
return
}
if err := localChecker.Run(); err != nil {
utils.CheckErr(errors.Wrap(err, "Failed to run preflight checker"))
}
Expand All @@ -51,6 +60,9 @@ func newCmdCheckPreflight(globalOpts *types.GlobalCmdOptions) *cobra.Command {
},

PostRun: func(cmd *cobra.Command, args []string) {
if localChecker.DataEngineMode == string(types.DataEngineModeV2Only) {
return
}
if err := localChecker.Output(); err != nil {
utils.CheckErr(errors.Wrap(err, "Failed to output preflight checker collection"))
}
Expand All @@ -61,6 +73,7 @@ func newCmdCheckPreflight(globalOpts *types.GlobalCmdOptions) *cobra.Command {

utils.SetGlobalOptionsLocal(cmd, globalOpts)

bindLocalDataEngineMode(cmd, &localChecker.DataEngineMode)
cmd.Flags().StringVarP(&localChecker.OutputFilePath, consts.CmdOptOutputFile, "o", os.Getenv(consts.EnvOutputFilePath), "Output the result to a file, default to stdout.")
cmd.Flags().BoolVar(&localChecker.EnableSpdk, consts.CmdOptEnableSpdk, utils.ConvertStringToTypeOrDefault(os.Getenv(consts.EnvEnableSpdk), false), "Enable checking of SPDK required packages, modules, and setup.")
cmd.Flags().IntVar(&localChecker.HugePageSize, consts.CmdOptHugePageSize, utils.ConvertStringToTypeOrDefault(os.Getenv(consts.EnvHugePageSize), 2048), "Specify the huge page size in MiB for SPDK.")
Expand Down
15 changes: 15 additions & 0 deletions cmd/local/subcmd/check_v2.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
package subcmd

import "errors"

type preflightReportRunner interface {
Run() error
Output() error
}

// Write the complete report before returning the qualification verdict. In the
// remote transport phase the checker itself suppresses only that final verdict.
func runAndOutputV2Preflight(checker preflightReportRunner) error {
runErr := checker.Run()
return errors.Join(runErr, checker.Output())
}
28 changes: 28 additions & 0 deletions cmd/local/subcmd/check_v2_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
package subcmd

import (
"errors"
"testing"
)

type failedPreflightReport struct {
runErr, outputErr error
wrote bool
}

func (report *failedPreflightReport) Run() error { return report.runErr }
func (report *failedPreflightReport) Output() error {
report.wrote = true
return report.outputErr
}

func TestV2PreflightFailureStillOutputsReport(t *testing.T) {
qualificationErr, outputErr := errors.New("host qualification failed"), errors.New("report write failed")
for _, writeErr := range []error{nil, outputErr} {
report := &failedPreflightReport{runErr: qualificationErr, outputErr: writeErr}
err := runAndOutputV2Preflight(report)
if !report.wrote || !errors.Is(err, qualificationErr) || (writeErr != nil && !errors.Is(err, writeErr)) {
t.Fatalf("missing report or failure: wrote=%v error=%v", report.wrote, err)
}
}
}
35 changes: 35 additions & 0 deletions cmd/local/subcmd/data_engine_mode.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
package subcmd

import (
"fmt"
"os"
"strconv"

"github.com/spf13/cobra"

"github.com/longhorn/cli/pkg/consts"
"github.com/longhorn/cli/pkg/types"
)

func bindLocalDataEngineMode(cmd *cobra.Command, target *string) {
value := os.Getenv(consts.EnvDataEngineMode)
if value == "" {
value = string(types.DataEngineModeV1Capable)
}
cmd.Flags().StringVar(target, consts.CmdOptDataEngineMode, value, "Installation capability: v1-capable or v2-only. V2-only omits iSCSI and requires SPDK qualification.")
}

func validateLocalDataEngineEnvironment(cmd *cobra.Command, value string) error {
mode, err := types.ParseDataEngineMode(value)
if err != nil {
return err
}
if mode == types.DataEngineModeV2Only && !cmd.Flags().Changed(consts.CmdOptHugePageSize) {
if raw, present := os.LookupEnv(consts.EnvHugePageSize); present {
if _, err := strconv.Atoi(raw); err != nil {
return fmt.Errorf("v2-only hugepage environment must contain an explicit integer")
}
}
}
return nil
}
23 changes: 20 additions & 3 deletions cmd/local/subcmd/install.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,15 @@ import (
"fmt"
"os"

"github.com/pkg/errors"
"github.com/sirupsen/logrus"
"github.com/spf13/cobra"

"github.com/pkg/errors"

"github.com/longhorn/cli/pkg/consts"
local "github.com/longhorn/cli/pkg/local/preflight"
"github.com/longhorn/cli/pkg/types"
"github.com/longhorn/cli/pkg/utils"

local "github.com/longhorn/cli/pkg/local/preflight"
)

func NewCmdInstall(globalOpts *types.GlobalCmdOptions) *cobra.Command {
Expand All @@ -38,14 +38,27 @@ func newCmdInstallPreflight(globalOpts *types.GlobalCmdOptions) *cobra.Command {
These dependencies ensure your Kubernetes cluster meets the requirements for successful Longhorn operation.`,

PreRun: func(cmd *cobra.Command, args []string) {
if err := validateLocalDataEngineEnvironment(cmd, localInstaller.DataEngineMode); err != nil {
utils.CheckErr(err)
}
localInstaller.LogLevel = globalOpts.LogLevel
localInstaller.AttemptID = os.Getenv(consts.EnvPreflightAttemptID)

if err := localInstaller.Init(); err != nil {
utils.CheckErr(errors.Wrap(err, "Failed to initialize preflight installer"))
}
},

Run: func(cmd *cobra.Command, args []string) {
if localInstaller.DataEngineMode == string(types.DataEngineModeV2Only) {
if os.Getenv(consts.EnvPreflightCollectionOnly) == "true" {
utils.CheckErr(localInstaller.RunForCollection())
utils.CheckErr(localInstaller.Output())
return
}
utils.CheckErr(runAndOutputV2Preflight(&localInstaller))
return
}
if err := localInstaller.Run(); err != nil {
utils.CheckErr(errors.Wrap(err, "Failed to run preflight installer"))
}
Expand All @@ -54,6 +67,9 @@ These dependencies ensure your Kubernetes cluster meets the requirements for suc
},

PostRun: func(cmd *cobra.Command, args []string) {
if localInstaller.DataEngineMode == string(types.DataEngineModeV2Only) {
return
}
if err := localInstaller.Output(); err != nil {
utils.CheckErr(errors.Wrap(err, "Failed to output preflight checker collection"))
}
Expand All @@ -64,6 +80,7 @@ These dependencies ensure your Kubernetes cluster meets the requirements for suc

utils.SetGlobalOptionsLocal(cmd, globalOpts)

bindLocalDataEngineMode(cmd, &localInstaller.DataEngineMode)
cmd.Flags().StringVarP(&localInstaller.OutputFilePath, consts.CmdOptOutputFile, "o", os.Getenv(consts.EnvOutputFilePath), "Output the result to a file, default to stdout.")
cmd.Flags().BoolVar(&localInstaller.UpdatePackages, consts.CmdOptUpdatePackages, utils.ConvertStringToTypeOrDefault(os.Getenv(consts.EnvUpdatePackageList), true), "Update packages before installing required dependencies.")
cmd.Flags().BoolVar(&localInstaller.EnableSpdk, consts.CmdOptEnableSpdk, utils.ConvertStringToTypeOrDefault(os.Getenv(consts.EnvEnableSpdk), false), "Enable installation of SPDK required packages, modules, and setup.")
Expand Down
22 changes: 20 additions & 2 deletions cmd/remote/subcmd/check.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
package subcmd

import (
stderrors "errors"

"github.com/pkg/errors"
"github.com/sirupsen/logrus"
"github.com/spf13/cobra"
Expand Down Expand Up @@ -58,6 +60,9 @@ INFO[2024-07-16T17:17:42+08:00] Cleaning up preflight checker
INFO[2024-07-16T17:17:42+08:00] Completed preflight checker`,

PreRun: func(cmd *cobra.Command, args []string) {
if err := preflightChecker.ValidateDataEngineMode(); err != nil {
utils.CheckErr(err)
}
preflightChecker.Image = globalOpts.Image
preflightChecker.ImageRegistry = globalOpts.ImageRegistry
preflightChecker.ImagePullSecret = globalOpts.ImagePullSecret
Expand All @@ -72,14 +77,23 @@ INFO[2024-07-16T17:17:42+08:00] Completed preflight checker`,
}

logrus.Info("Cleaning up preflight checker")
if err := preflightChecker.Cleanup(); err != nil {
utils.CheckErr(errors.Wrapf(err, "Failed to cleanup preflight checker"))
if preflightChecker.DataEngineMode != string(types.DataEngineModeV2Only) {
if err := preflightChecker.Cleanup(); err != nil {
utils.CheckErr(errors.Wrapf(err, "Failed to cleanup preflight checker"))
}
}
},

Run: func(cmd *cobra.Command, args []string) {
logrus.Info("Running preflight checker")
output, err := preflightChecker.Run()
if preflightChecker.DataEngineMode == string(types.DataEngineModeV2Only) {
if output != "" {
logrus.Infof("Retrieved preflight checker result:\n%v", output)
}
utils.CheckErr(stderrors.Join(err, preflightChecker.Cleanup()))
return
}
if err != nil {
utils.CheckErr(errors.Wrap(err, "Failed to run preflight checker"))
}
Expand All @@ -88,6 +102,9 @@ INFO[2024-07-16T17:17:42+08:00] Completed preflight checker`,
},

PostRun: func(cmd *cobra.Command, args []string) {
if preflightChecker.DataEngineMode == string(types.DataEngineModeV2Only) {
return
}
logrus.Info("Cleaning up preflight checker")
if err := preflightChecker.Cleanup(); err != nil {
utils.CheckErr(errors.Wrapf(err, "Failed to cleanup preflight checker"))
Expand All @@ -99,6 +116,7 @@ INFO[2024-07-16T17:17:42+08:00] Completed preflight checker`,

utils.SetGlobalOptionsRemote(cmd, globalOpts)

cmd.Flags().StringVar(&preflightChecker.DataEngineMode, consts.CmdOptDataEngineMode, string(types.DataEngineModeV1Capable), "Installation capability: v1-capable or v2-only. V2-only omits iSCSI and requires SPDK qualification.")
cmd.Flags().BoolVar(&preflightChecker.EnableSpdk, consts.CmdOptEnableSpdk, false, "Enable checking of SPDK required packages, modules, and setup.")
cmd.Flags().IntVar(&preflightChecker.HugePageSize, consts.CmdOptHugePageSize, 2048, "Specify the huge page size in MiB for SPDK.")
cmd.Flags().StringVar(&preflightChecker.UserspaceDriver, consts.CmdOptUserspaceDriver, "", "Userspace I/O driver for SPDK.")
Expand Down
23 changes: 21 additions & 2 deletions cmd/remote/subcmd/install.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ package subcmd
import (
"fmt"

stderrors "errors"

"github.com/pkg/errors"
"github.com/sirupsen/logrus"
"github.com/spf13/cobra"
Expand Down Expand Up @@ -64,6 +66,9 @@ If a reboot is required, the following message will be displayed:
- Need to reboot the system and execute longhornctl install preflight again`,

PreRun: func(cmd *cobra.Command, args []string) {
if err := preflightInstaller.ValidateDataEngineMode(); err != nil {
utils.CheckErr(err)
}
preflightInstaller.Image = globalOpts.Image
preflightInstaller.ImageRegistry = globalOpts.ImageRegistry
preflightInstaller.ImagePullSecret = globalOpts.ImagePullSecret
Expand All @@ -79,14 +84,24 @@ If a reboot is required, the following message will be displayed:
}

logrus.Info("Cleaning up preflight installer")
if err := preflightInstaller.Cleanup(); err != nil {
utils.CheckErr(errors.Wrapf(err, "Failed to cleanup preflight installer"))
if preflightInstaller.DataEngineMode != string(types.DataEngineModeV2Only) {
if err := preflightInstaller.Cleanup(); err != nil {
utils.CheckErr(errors.Wrapf(err, "Failed to cleanup preflight installer"))
}
}
},

Run: func(cmd *cobra.Command, args []string) {
logrus.Info("Running preflight installer")
output, err := preflightInstaller.Run()
if preflightInstaller.DataEngineMode == string(types.DataEngineModeV2Only) {
if output != "" {
logrus.Infof("Retrieved preflight installer result:\n%v", output)
}
utils.CheckErr(stderrors.Join(err, preflightInstaller.Cleanup()))
logrus.Info("Completed V2 prerequisite setup; run check preflight --data-engine-mode=v2-only to qualify every host")
return
}
if err != nil {
utils.CheckErr(errors.Wrap(err, "Failed to run preflight installer"))
}
Expand All @@ -95,6 +110,9 @@ If a reboot is required, the following message will be displayed:
},

PostRun: func(cmd *cobra.Command, args []string) {
if preflightInstaller.DataEngineMode == string(types.DataEngineModeV2Only) {
return
}
logrus.Info("Cleaning up preflight installer")
if preflightInstaller.OperatingSystem != string(consts.OperatingSystemContainerOptimizedOS) {
if err := preflightInstaller.Cleanup(); err != nil {
Expand All @@ -110,6 +128,7 @@ If a reboot is required, the following message will be displayed:

utils.SetGlobalOptionsRemote(cmd, globalOpts)

cmd.Flags().StringVar(&preflightInstaller.DataEngineMode, consts.CmdOptDataEngineMode, string(types.DataEngineModeV1Capable), "Installation capability: v1-capable or v2-only. V2-only omits iSCSI and requires SPDK setup.")
cmd.Flags().StringVar(&preflightInstaller.OperatingSystem, consts.CmdOptOperatingSystem, "", "Specify the operating system (\"\", cos). Leave this empty to use the package manager for installation.")
cmd.Flags().BoolVar(&preflightInstaller.UpdatePackages, consts.CmdOptUpdatePackages, true, "Update packages before installing required dependencies.")
cmd.Flags().BoolVar(&preflightInstaller.EnableSpdk, consts.CmdOptEnableSpdk, false, "Enable installation of SPDK required packages, modules, and setup.")
Expand Down
2 changes: 2 additions & 0 deletions deploy/check.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ spec:
securityContext:
privileged: true
env:
- name: DATA_ENGINE_MODE
value: "v1-capable"
- name: ENABLE_SPDK
value: "true"
- name: HUGEMEM
Expand Down
2 changes: 2 additions & 0 deletions deploy/install.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ spec:
securityContext:
privileged: true
env:
- name: DATA_ENGINE_MODE
value: "v1-capable"
- name: UPDATE_PACKAGE_LIST
value: "true"
- name: ENABLE_SPDK
Expand Down
4 changes: 2 additions & 2 deletions docs/longhornctl.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ A CLI tool for troubleshooting and managing Longhorn operations.

```
-h, --help help for longhornctl
--image string Image containing longhornctl-local (default "longhornio/longhorn-cli:v1.13.0-dev")
--image string Image containing longhornctl-local (default "longhornio/longhorn-cli:v1.14.0-dev")
--image-pull-secret string Secret with registry credentials for pulling images
--image-registry string Registry to apply to all images (CLI, engine, pause, BCI, etc.), replacing any registry already specified in those images.
--kubeconfig string Kubernetes config (kubeconfig) path
Expand All @@ -33,4 +33,4 @@ A CLI tool for troubleshooting and managing Longhorn operations.
* [longhornctl trim](longhornctl_trim.md) - Longhorn trimming operations
* [longhornctl version](longhornctl_version.md) - Print longhornctl version

###### Auto generated by spf13/cobra on 6-Jul-2026
###### Auto generated by spf13/cobra on 2-Nov-2026
Loading
Loading