Issue #8285 : Add Hadoop-free HDFS VFS plugin (WebHDFS/HttpFS/Knox) - #8287
Open
mattcasters wants to merge 12 commits into
Open
Issue #8285 : Add Hadoop-free HDFS VFS plugin (WebHDFS/HttpFS/Knox)#8287mattcasters wants to merge 12 commits into
mattcasters wants to merge 12 commits into
Conversation
…nox) Named HDFS connections talk REST (HttpFS, Knox, or WebHDFS) with JDK Kerberos/SPNEGO and keytab re-login. No Hadoop client JARs, so Parquet and other VFS writers can reach CDP 7.1.8 and 7.3.2 from OpenShift without matching cluster libraries. Marketplace-only; does not register a global hdfs:// scheme.
Keep both how-to nav entries: HDFS/OpenShift from this branch and scheduling from apache#8273.
…ection Each Cluster, Kerberos and TLS tab gets a Test button that probes the values currently in the editor. TLS trust material accepts PEM/CRT CA bundles as well as JKS/PKCS12 keystores.
…ugin classloaders GuiPluginType ignored classLoaderGroup, so the editor buttons ran against a second copy of HdfsMeta. Share vfs-hdfs with the GUI plugin and invoke the button method on the live editor object.
Contributor
Author
|
With TLS and Kerberos working I got the following error testing the WebHDFS endpoint: Detailed stack trace: |
… to forward slashes JAAS treats backslash as an escape, so a Browse result like C:\Users\... breaks Kerberos login. Store and use C:/Users/... instead.
…er test steps The Cluster Test button logged in, then built the Negotiate token on the calling thread. With useSubjectCredsOnly the GSS layer never saw the TGT (Failed to find any Kerberos tgt). Token creation now runs in doAs, and the Test dialog lists TLS, Kerberos, SPNEGO and GETFILESTATUS so a later failure names the step that actually broke.
… NameNode GETFILESTATUS 403 "state standby" means the WebHDFS host is the HA standby. Endpoint hostname now takes Impala-style comma-separated hosts (master1,master2) and retries the next one. Test dialogs use the title "Test results" so the window no longer stretches off-screen.
mattcasters
marked this pull request as ready for review
September 9, 2026 10:44
…ve NameNode OPEN now uses noredirect and fetches the DataNode Location, matching CREATE, so an empty 307 is not treated as the file. The HTTP client disables gzip and automatic redirects. After HA failover the working NameNode is tried first, and standby messages are logDetailed.
mattcasters
marked this pull request as draft
September 9, 2026 12:53
Contributor
Author
|
Found an issue when writing to HDFS: |
Contributor
Author
|
And a StackOverflowException: |
GZIPInputStream throws EOFException when an HTTP/VFS body ends a few bytes short of the CRC trailer. Map that to -1 so Text File Input can finish a large gzip CSV, and buffer the inflater and HDFS entity reads.
doGetType no longer calls getType (StackOverflowError with multiple Text File Output copies). doAttach no longer marks the file attached before GETFILESTATUS. A 404 from the active NameNode is remembered so CREATE does not go back to the standby. Only standby and unreachable hosts fail over.
mattcasters
marked this pull request as ready for review
September 9, 2026 13:24
mattcasters
marked this pull request as draft
September 9, 2026 13:31
Contributor
Author
|
Looks like CRUD is working. The parallel write/read/... edge cases are being put into extra integration tests. |
… in ITs Extend integration-tests/hadoop with gzip CSV write/read, four-copy Text File Output, parquet read-back, and a WebHDFS connection whose first NameNode is unreachable so write/read must fail over.
mattcasters
marked this pull request as ready for review
September 9, 2026 13:47
Contributor
Author
Security & Integrity Review for PR #8287 / Issue #8285Overall, moving to a Hadoop-free WebHDFS / HttpFS / Knox VFS provider is a huge security win—it eliminates hundreds of transitive dependencies, outdated RPC libraries, and common CVEs. Hostname verification is enabled by default, SPNEGO credential delegation is disabled ( Here is a summary of security, data integrity, and credential-handling findings to address: 1. [Critical / Data Integrity] Silent Data Truncation in
|
mattcasters
marked this pull request as draft
September 10, 2026 08:21
…beros) Stop treating truncated gzip as success. Mint SPNEGO only for configured gateway hosts. Decrypt truststore passwords. Reject HTTP DataNode Locations on HTTPS connections (opt-in checkbox). Serialize JVM Kerberos config, ignore Realm/KDC when krb5.conf is set, and unregister sessions when tests and filesystems close.
Contributor
Author
|
Addressed the security review:
Rebuild |
Contributor
Author
Follow-up Review: Security & Integrity Recommendations Addressed in
|
mattcasters
marked this pull request as ready for review
September 10, 2026 08:36
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a marketplace HDFS VFS plugin so Hop pipelines (including Parquet File Output) can write to HDFS without shipping Hadoop client JARs.
Parquet already writes through
HopVfs.getOutputStream(). Matching Cloudera parcel libraries in Hop would fight the Hadoop 3.4.2 bits already on the classpath and would break on a 7.1.8 (Hadoop 3.1.1) → 7.3.2 (Hadoop 3.4.2) upgrade. This plugin talks the stable WebHDFS REST API instead.What changed
hop-tech-hadoopmarketplace plugin. Named VFS connection (scheme is the connection name, e.g.cdp:///warehouse/...). Does not register a globalhdfs://provider — Native Spark already uses that as a Hadoop FileSystem URI.kinitbinary, noUserGroupInformation.Tests
integration-tests/hadoop(unsecured WebHDFS + text/Parquet). Not run against a live cluster in this change.OpenShift / CDP note
From a remote container, use HttpFS or Knox. Raw WebHDFS and native HDFS RPC both need DataNode ports from the pod.
Fixes #8285