Stop changing the case for COPY TO option values

@xinlifoobar

Describe the bug

Currently, we're changing the case of every option value passed with the COPY statement:

datafusion/datafusion/sql/src/statement.rs

Lines 891 to 900 in 3773fb7

    
               if !(&key.contains('.')) { 
        
                   // If config does not belong to any namespace, assume it is 
        
                   // a format option and apply the format prefix for backwards 
        
                   // compatibility. 
        
                   let renamed_key = format!("format.{}", key); 
        
                   options.insert(renamed_key.to_lowercase(), value_string.to_lowercase()); 
        
               } else { 
        
                   options.insert(key.to_lowercase(), value_string.to_lowercase()); 
        
               } 
        
           }

This causes an issue in cases where the option values for some external integrations might be case sensitive (eg. like access keys).

@xinlifoobar found this while working on adding an integration to huggingface in #10792.

See previous discussion here: #9723 (comment)

To Reproduce

No response

Expected behavior

There should be some way to pass option values as is for certain integrations.

Additional context

No response

@metesynnada

fyi @metesynnada

Sorry if I mislead you. I encountered this issue in the CREATE EXTERNAL TABLE statement as

datafusion/datafusion/sql/src/statement.rs

Lines 1017 to 1025 in 3773fb7

    
           if !(&key.contains('.')) { 
        
               // If a config does not belong to any namespace, we assume it is 
        
               // a format option and apply the format prefix for backwards 
        
               // compatibility. 
        
               let renamed_key = format!("format.{}", key.to_lowercase()); 
        
               options_map.insert(renamed_key, value_string.to_lowercase()); 
        
           } else { 
        
               options_map.insert(key.to_lowercase(), value_string.to_lowercase()); 
        
           }

How has such an issue typically been resolved previously? If adding a ParseOptions, e.g., enable_options_value_normalization, is a reasonable way to fix it, I could help there.

datafusion/datafusion/sql/src/planner.rs

Lines 97 to 100 in 47026a2

    
           pub struct ParserOptions { 
        
               pub parse_float_as_decimal: bool, 
        
               pub enable_ident_normalization: bool, 
        
           }

I believe we can remove the lowercase conversion here and instead add it when we set ConfigField's (via alter_with_string_hash_map) for the cases where we can ignore case sensitivity.

If no one else is working on this issue, I'll take it. My plan is to remove the automatic conversion of values to lowercase and to create a lookup table for keys. This table will indicate the specific standardization methods.

Another potential option is to follow the model of the datafusion.sql_parser.enable_ident_normalization configuration setting to control if the options should be case sensitive or not. That might be the "least surprising" to people who are used (or not used) to SQL identifier normalization rules

I don't remember wll the usecases for configuration option case sensitivity, so please ignore this suggestion if it doesn't make sense

enable_options_value_normalization

I draft #11330 to discuss this. Let me know your thoughts!

Navigation Menu

Repository navigation

Stop changing the case for COPY TO option values #10853

Describe the bug

To Reproduce

Expected behavior

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	if !(&key.contains('.')) {
	// If config does not belong to any namespace, assume it is
	// a format option and apply the format prefix for backwards
	// compatibility.
	let renamed_key = format!("format.{}", key);
	options.insert(renamed_key.to_lowercase(), value_string.to_lowercase());
	} else {
	options.insert(key.to_lowercase(), value_string.to_lowercase());
	}
	}

	if !(&key.contains('.')) {
	// If a config does not belong to any namespace, we assume it is
	// a format option and apply the format prefix for backwards
	// compatibility.
	let renamed_key = format!("format.{}", key.to_lowercase());
	options_map.insert(renamed_key, value_string.to_lowercase());
	} else {
	options_map.insert(key.to_lowercase(), value_string.to_lowercase());
	}

	pub struct ParserOptions {
	pub parse_float_as_decimal: bool,
	pub enable_ident_normalization: bool,
	}

Stop changing the case for COPY TO option values #10853

Description

Describe the bug

To Reproduce

Expected behavior

Additional context

Activity

alamb commented on Jun 10, 2024

xinlifoobar commented on Jun 11, 2024

berkaysynnada commented on Jun 11, 2024

berkaysynnada commented on Jul 3, 2024

alamb commented on Jul 5, 2024

xinlifoobar commented on Jul 8, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions