Big Query Create or Replace table mysteriously fails when clustering or partitioning is changed

This Big Query script will fail

create or replace table joel_dataset.demo_table (
  int_field INT64,
  int_field2 INT64,
  changed_time TIMESTAMP,)
partition by range_bucket(
            int_field,
            generate_array(0, 10, 1)
        )
;

create or replace table joel_dataset.demo_table (
  int_field INT64,
  int_field2 INT64,
  changed_time TIMESTAMP,)
partition by range_bucket(
            int_field,
            generate_array(0, 10, 1)
        )
cluster by int_field2 --adding clustering
;


Invalid value: Incompatible table partitioning specification. Expected partitioning specification range(type:integer,field:int_field,start:0,end:10,interval:1) clustering(int_field2), but input partitioning specification is range(type:integer,field:int_field,start:0,end:10,interval:1) at [2:1]

The reason seem to be that "or replace" does not work. Big Query evaluates any preexisting table and crashes if you change anything.
It will also fail with mysterious erro if a view exists with the same name.

This Big Query script will work


drop table if exists joel_dataset.demo_table;
create or replace table joel_dataset.demo_table (
  int_field INT64,
  int_field2 INT64,
  changed_time TIMESTAMP,)
partition by range_bucket(
            int_field,
            generate_array(0, 10, 1)
        )
;

drop table if exists joel_dataset.demo_table;

create or replace table joel_dataset.demo_table (
  int_field INT64,
  int_field2 INT64,
  changed_time TIMESTAMP,)
partition by range_bucket(
            int_field,
            generate_array(0, 10, 1)
        )
cluster by int_field2
;

Comments

Post a Comment

Popular posts from this blog

SSIS: Set parent package variable from child package

SSIS: Set parent package variable from child package For this sample we communicate using bool value named ReturnValue First create a variable in parent package: ReturnValue as Boolean In child package, you do not need to set package configuration for the parent variable, nor should the ReturnValue variable be created. Create script task in child package. Leave the fields ReadOnlyVariables, ReadWriteVariables empty. public void Main() { Variables vars = null ; Dts.VariableDispenser.GetVariables( ref vars); var dummy = false ; if (Dts.VariableDispenser.Contains( "User::ReturnValue" )) { Dts.Events.FireInformation(0, "" , "Setting (parent package variable) User::ReturnValue" , null , 0, ref dummy); Dts.VariableDispenser.LockForWrite( "User::ReturnValue" ); Dts.VariableDispenser.GetVariables( ref vars); vars[ "User::ReturnValue" ].Value = true ; ...
Read more

How to decrypt stored password from SSMS registered servers

Image
Right click your server, choose tasks/export Uncheck "Do not include user names and passwords in the export file" Save the export to a file Open the file in an editor and locate the tag RegisteredServers:ConnectionStringWithEncryptedPassword Copy the contents of password attribute Open powershell and paste the code below, after you amend it with your encrypted string. $base64pass = "PasteEncryptedPasswordHere" [System.Reflection.Assembly]::LoadWithPartialName("System.Security") | Out-Null [System.Text.Encoding]::Unicode.GetString([System.Security.Cryptography.ProtectedData]::Unprotect([System.Convert]::FromBase64String($base64pass), $null, [System.Security.Cryptography.DataProtectionScope]::CurrentUser))
Read more

Insert bulk statement does not support recompile (SQL 2017)

This happened to me once, source and target databases was located on the same server running 14.0.1000.169 Microsoft SQL Server Enterprise (64-bit) Consecutive executions never fail. I could not reproduce the error until next day. My workaround has been to simply add "re-try execution" a number of times in SQL Agent job. SSIS Error Code DTS_E_OLEDBERROR.  An OLE DB error has occurred. Error code: 0x80004005.  An OLE DB record is available.  Source: "Microsoft SQL Server Native Client 11.0" Hresult: 0x80004005  Description: "Insert bulk statement does not support recompile.".  End Error
Read more