All our scheduled queries have been failing with `Error code 3 : Incompatible table partitioning specification.` since 2019-04-23 at 8PM UTC - google-bigquery

Our scheduled queries have been running for months without any hiccups, but starting from 8pm UTC on 2019-04-23, they failed with the following error, and they are still failing very often 36 hours later.
11:00:01 PM Error code 3 : Incompatible table partitioning specification. Destination table exists with partitioning specification interval(type:DAY,field:), but transfer target partitioning specification is interval(type:DAY,field:). Please retry after updating either the destination table or the transfer partitioning specification.
11:00:00 PM Starting to process the query job with parameter #run_date=2019-04-23.
11:00:00 PM Dispatched run to data source with id 538824528883320
The following screenshot shows that some runs are ok (but none of our queries had successful runs today):
We tried redeploying the queries, but they still fail on the first run. Hitting Retry generates the same error too.

This is a known issue and should be fixed soon.


How to debug specific SQL occasionally result in BigQuery “Request timed out. Please try again”

In the last week or so a subset (single digit out of thousands per day) of the SQL we submit to BigQuery in interactive mode started to take hours instead of seconds. The SQL for the jobs that timed out appeared to very specific cases. I was able to reproduce the behavior with these two jobs from the BigQuery console:
The working invocation (ran in 5 secs):
Job ID bluecore-qa:US.bquijob_4e0e4662_1639a278fcf
Creation Time May 25, 2018, 9:54:34 PM
Start Time May 25, 2018, 9:54:34 PM
End Time May 25, 2018, 9:54:39 PM
Bytes Processed 176 MB
Bytes Billed 177 MB
Slot Time (ms) 271 K
The exact same SQL (ran less than a minute later) that timed out after 6 hours:
Job ID bluecore-qa:US.bquijob_57c799e2_1639a2852fa
Creation Time May 25, 2018, 9:55:24 PM
Start Time May 25, 2018, 9:55:24 PM
Query Priority Interactive
Job Type State Start Time Duration User Email Bytes Processed Bytes Billed Billing Tier Labels
---------- --------- ----------------- ---------- --------------------- ----------------- -------------- -------------- --------
query FAILURE 25 May 21:55:24 5:59:45 xxxxx
Error encountered during job execution:
Request timed out. Please try again.
Note that the SQL does use 'IGNORE CASE' which has been problematic for us in the past (but usually resulting in an 'internal error' case).
Is there a way to get more information on the job to see whether the 2nd job got pushed back in the BigQuery scheduling queue?
(According to BigQuery StackDriver stats we stay well under the 2000 slot limit for our project).
In BigQuery, jobs never pushed back to the BigQuery scheduling queue (even in streaming and interactive modes).
You can use Audit logs to get more details about the timeouts. Audit logging is currently the only alternative available for the Stackdriver.
This is a Google BigQuery side issue. They are working on a fix and will update once this is out.

Google BigQuery Create/append to table from Avro internalError

I am fairly new to BigQuery, however I have been able to create and append to existing BigQuery tables from Avro files (both in EU region) until 1-2 days ago. I am only using the web UI so far.
I just attempted to create a new table from a newly generated Avro file and got the same error, details below:
Job ID bquijob_670fd977_15655fb3da1
Start Time Aug 4, 2016, 3:35:45 PM
End Time Aug 4, 2016, 3:35:53 PM
Write Preference Write if empty
An internal error occurred and the request could not be completed.
(error code: internalError)
I am unable to debug because there is not really anything to go by.
We've just released a new feature to not creating the root field:
Since you have imported Avro before, we have excluded your project from this new feature. But unfortunately we had a bug with exclusion, and will cause reading Avro to fail. I think you most likely ran into this problem.
The fix will be released next week. If you don't need the root field and want to enable your project for the new feature, please send the project id to me, huazhang at Sorry for the trouble this has caused.

BigQuery getting resource exceeded during query execution issue on query that worked yesterday

I have a very unusual situation that arose over the last 48 hours. I am dealing with a BigQuery SQL query that is currently getting a "Resources exceeded during execution" error. The query has run successfully on a CRON for a number of months and died about 48 hours ago. Below is a snapshot of the query:
The job ID is shown above, and is: model-folio-818.job_dEdyeaHonw2PWz2z0jVVgBekGsg
Has anyone noticed anything unusual over the last few days that has impacted their queries? The BigQuery instance is currently running the production data warehouse environment and hence the breakdown is causing major angst. Any insight from anyone here on similar changes or the Google team would be greatly appreciated!

Changes in query behaviour

I have some queries that run every day for several month with no problem. I didn't change anything in the queries for a long while.
In the past few days some of them fail. Error message says something regarding some fields: "Field 'myfield' not found.". these queries usually involve some sub-queries and window functions.
Example for the BQ guys:
On 2015-08-03 Job ID: job_EUWyK5DIFSxJxGAEC4En2Q_hNO8 run successfully
on the following days, same query, failed. Job IDs: (job_A9KYJLbQJQvHjh1g7Fc0Abd2qsc , job__15ff66aYseR-YjYnPqWmSJ30N8)
In addition, for some other queries running times extended from minutes to hours and sometime return "timeout".
My questions:
Was something changed in the BQ engine?
What should I do to make my queries run again?
So the problem could be two folded:
An update to query execution engine was rolled out during the week of August 3, 2015 as documented in the announcement
If this is the case, you need to update your queries.
Some performance issues were detected lately, but in order to actually identify if there is something wrong with your project or not, you need to create a issue request I did the same in past and it was fixed.

Getting “Backend error. Job aborted” while trying to export a big query table to GCS

Since the past couple of weeks I have been continuously getting "Backend error. Job aborted" error while trying to export a big query table to google cloud storage in csv format.
The table has been created using bq select * statement (using allowLargeQueryResult option)
Also the target bucket name doesn't seem to be problematic.
Here's a sample extract.
Backend error. Job aborted.
Job ID:
Start Time: 2:53pm, 8 Aug 2014
End Time: 8:53pm, 8 Aug 2014
The job is taking almost six hours to complete after which it fails. Previously it used to complete in a couple of minutes. Any help would be appreciated.
Your export job hit a timeout. We're currently investigating why; the date of your job coincides with a bandwidth issue we were having that should have been resolved. We're currently adding more instrumentation and monitoring so it will be easier to debug in the future.
As a workaround, if you give multiple extraction URI patterns, BigQuery will spin up more workers in parallel. See the "Multiple Wildcard URIs" example here.
As Jordan said, this coincided with a bandwidth problem. Sorry for the inconvenience.
In some cases, giving multiple wildcard URIs will increase parallelism, but this applies only to fairly large (10's of GB) tables, and can actually decrease parallelism. Multiple wildcard URIs are designed to support Hadoop jobs, not to control parallelism.