columnStoreExporter Module¶
Functions¶
generateTableStatement¶
-
columnStoreExporter.
generateTableStatement
(dataFrame, database=None, table="spark_export", determineTypeLengths=False)¶ Generates a CREATE TABLE statement based on the schema of the submitted DataFrame.
Returns: A DML CREATE TABLE statement based on the schema of the submitted DataFrame.
Parameters: - dataFrame – The DataFrame from whom the structure for the generated table statement will be inferred.
- database – The database name used in the generated table statement.
- table – The table name used in the generated table statement.
- determineTypeLength – If set to True the content DataFrame will be analysed to determine the best SQL datatype for each column. Otherwise reasonable default types will be used.
Note
The submitted database and table names will automatically be parsed into the ColumnStore naming convention, if not already compatible.
export¶
-
columnStoreExporter.
export
(database, table, df, configuration=None)¶ Exports the given DataFrame into an existing ColumnStore table.
Parameters: - database – The target database the DataFrame is exported into.
- table – The target table the DataFrame is exported into.
- df – The DataFrame to export.
- configuration – Path to the Columnstore.xml configuration to use for the export. If None is given, the default Columnstore.xml will be used.
Note
To guarantee that the DataFrame import into ColumnStore is a single transaction, that is rollbacked in case of error, the DataFrame is first collected at the Spark master and from there written to the ColumnStore system. Therefore, it needs to fit into the memory of the Spark master.
Note
The schema of the DataFrame to export and the ColumnStore table to import have to match. Otherwise, the import will fail.
parseTableColumnNameToCSConvention¶
-
columnStoreExporter.
parseTableColumnNameToCSConvention
(input)¶ Parses the input String according to the ColumnStore naming convention and returns it.
Returns: The parsed input String in ColumnStore naming convention. Parameters: input – The String that is going to be parsed.