How to bulk import NUL (0x0, ascii 0) character from csv to BigQuery

If you ever exported a csv file from MsSql via bcp you probably got a csv file filled with NUL chars




When using bq load you get this error 

BigQuery error in load operation: Error processing job '[project]:[job_id]': Error while reading data, error message: Error detected while parsing row starting at position: 0.
Error: Bad character (ASCII 0) encountered. File: gs://[bucket]/[path]/[file].csv
Failure details:
- Error while reading data, error message: CSV processing encountered

One solution is to let bq load these characters using this flag

--preserve_ascii_control_characters=true

Note that the unicode general "unknown character" � symbol is inserted instead of null and you may want to clean this post bulk. But at least bq load does not crash.




Comments

Post a Comment

Popular posts from this blog

SSIS: Set parent package variable from child package

SSIS: Set parent package variable from child package For this sample we communicate using bool value named ReturnValue First create a variable in parent package: ReturnValue as Boolean In child package, you do not need to set package configuration for the parent variable, nor should the ReturnValue variable be created. Create script task in child package. Leave the fields ReadOnlyVariables, ReadWriteVariables empty. public void Main() { Variables vars = null ; Dts.VariableDispenser.GetVariables( ref vars); var dummy = false ; if (Dts.VariableDispenser.Contains( "User::ReturnValue" )) { Dts.Events.FireInformation(0, "" , "Setting (parent package variable) User::ReturnValue" , null , 0, ref dummy); Dts.VariableDispenser.LockForWrite( "User::ReturnValue" ); Dts.VariableDispenser.GetVariables( ref vars); vars[ "User::ReturnValue" ].Value = true ; ...
Read more

How to decrypt stored password from SSMS registered servers

Image
Right click your server, choose tasks/export Uncheck "Do not include user names and passwords in the export file" Save the export to a file Open the file in an editor and locate the tag RegisteredServers:ConnectionStringWithEncryptedPassword Copy the contents of password attribute Open powershell and paste the code below, after you amend it with your encrypted string. $base64pass = "PasteEncryptedPasswordHere" [System.Reflection.Assembly]::LoadWithPartialName("System.Security") | Out-Null [System.Text.Encoding]::Unicode.GetString([System.Security.Cryptography.ProtectedData]::Unprotect([System.Convert]::FromBase64String($base64pass), $null, [System.Security.Cryptography.DataProtectionScope]::CurrentUser))
Read more

Insert bulk statement does not support recompile (SQL 2017)

This happened to me once, source and target databases was located on the same server running 14.0.1000.169 Microsoft SQL Server Enterprise (64-bit) Consecutive executions never fail. I could not reproduce the error until next day. My workaround has been to simply add "re-try execution" a number of times in SQL Agent job. SSIS Error Code DTS_E_OLEDBERROR.  An OLE DB error has occurred. Error code: 0x80004005.  An OLE DB record is available.  Source: "Microsoft SQL Server Native Client 11.0" Hresult: 0x80004005  Description: "Insert bulk statement does not support recompile.".  End Error
Read more