Data Collection

Data Engineering

AE DataOps

Operate the AE data-development platform through ae-cli, data warehouse management, flow orchestration, IDE SQL queries, and data integration and sync, upgrading data engineering from manual web operations to command-line-driven end-to-end workflows.

IndustriesAll industries

Agent · Skills Live

skill · loadedindustry+100

Run this Skill on my data

Query console

generated

query.sql · generated

SELECT channel, count(*)

FROM v_payment

GROUP BY channel — 0.4s

12 rows returned

Analysis readyevidence attached

The AE DataOps Skill is a data-development-and-operations capability in ThinkingAI's industry Skill library. It helps data engineering teams operate the AE data-development platform through the ae-cli command-line tool, space discovery, data table and view management, flow orchestration (create/schedule/publish/monitor), IDE SQL queries, and data integration and sync, with six modules covering the full data-development lifecycle, upgrading from manually handling data tasks one by one on the web to an efficient command-line-driven workflow. This is ThinkingAI's decade of industry experience, packaged into ready-to-use Skills.

Live walkthrough

AgentSave as AgentExport session

ae-dataops

Our reports have been getting slower and slower, and I have a feeling the 'paying users in the last 30 days' logic is being recomputed all over the place. Scan how many places actually repeat it, consolidate what should be consolidated into an intermediate table, and set up a daily auto-updating task flow.

Scan all report SQL in the workspace from the IDE to find repeated logic

The 'paying users in the last 30 days' CTE is repeated across 13 reports

Assess the bloat impact: trace the runtime of downstream tasks that reference this CTE

The tag computation has bloated from 30 minutes to 4 hours 2 minutes

Create the incremental intermediate table dws_pay_user_30d in DEV, refreshing only the day's new payment detail

Create the task flow flow_pay_user_30d with CRON 0 0 2 * * ? (daily 02:00)

Preview-publish to PROD and manually trigger the first run

First run succeeded in 4 minutes 12 seconds

Rewrite the 3 most-queried reports to reference the intermediate table and back-test

Single-report query 2–4 minutes → 5–8 seconds

The cleanup is done, and the root cause was more concentrated than expected. The 'paying users in the last 30 days' CTE was computed independently by 13 reports; it's now consolidated into the intermediate table dws_pay_user_30d with a daily task flow attached: the rewritten reports' queries dropped from 2–4 minutes to 5–8 seconds, and once the tag computation switches over it should fall from 4 hours back to under 15 minutes.

1. Task flow status

Task flow	Schedule	Last run	Duration	Status
flow_pay_user_30d (new this run)	Daily 02:00 (0 0 2 * * ?)	Manual first run today 14:32	4 min 12 sec	Live in PROD
flow_user_tag (tag computation)	Daily 03:00	Today 03:00	4 hr 2 min	Reference rewritten, effective tomorrow
flow_mysql_sync (data sync)	Daily 01:00	Today 01:00	11 min 40 sec	Normal
flow_report_ads (report processing)	Daily 05:00	Today 05:00	26 min 18 sec	Reference rewritten, effective tomorrow

← Scroll to see more

2. Cleanup actions this run

Scanned all report SQL in the workspace and confirmed the 'paying users in the last 30 days' CTE is repeated across 13 reports: the same definition is fully recomputed 13 times a day, the shared root cause of the slowness

Built the incremental intermediate table dws_pay_user_30d in DEV: it processes only the day's new payment detail, no longer recomputing the full 30-day window

Created the task flow with a 6-field CRON 0 0 2 * * ?, preview-published to PROD, and the first run succeeded in 4 minutes 12 seconds

Rewrote the references in the 3 most-queried reports and the tag task flow; the remaining 10 reports are listed on a migration checklist, best completed in two batches by query frequency

Conclusion

The essence of this cleanup is collapsing the same business definition scattered across 13 places into a single data asset that's scheduled, monitored, and has one source of truth. For future reports that need the paying-user definition, reference dws_pay_user_30d directly instead of hand-writing the CTE: definition drift is harder to debug than a slow query.

The flow_pay_user_30d task flow is live, auto-updates daily at 02:00, and failure alerts are wired into ops monitoring.

On your data

That was a simulated run

Leave your work email and we will run a live walkthrough on your real business data.

The problem

Data development is the foundational engineering of any data system, but over 60% of data engineering teams hit efficiency bottlenecks: creating a data flow requires manually configuring SQL task nodes, dependencies, schedule rules, and CRON expressions, taking 30 minutes on average; post-execution operations are even harder, with a failed task taking 15-20 minutes on average to troubleshoot; and data-integration configuration requires hand-writing JSON with scattered docs, with an error rate over 40%.

What it does

Full coverage of six modules: repo (space discovery) + datatable (table/view management) + flow (flow orchestration) + operations (ops monitoring) + ide (SQL queries) + integration (data integration)

Six scenario routes: flow creation, flow monitoring, data integration, data query, table management, and space discovery, automatically matched to user intent

Complete flow lifecycle: DEV configuration to preview-publish to PROD manual execution to ops troubleshooting, with a command for each step

Clear ID system: executeId (early-stop handle) vs flowInstanceId (stable ops handle), with different IDs for different scenarios

CRON 6-field spec: second minute hour day month weekday (one more "second" field than the standard 5), with common templates ready to use

When to use it

Creating a data flow (adding SQL task nodes, configuring dependencies and scheduling, publishing to PROD)

Monitoring flow execution status, viewing task logs, and stopping abnormal instances

Creating a data source and configuring a data-sync plan

Running SQL queries in the IDE or browsing data table structures

Creating a data table or view and publishing it to PROD

Discovering available DataOps spaces and space codes

In the field

Case

A data engineering team · batch creation of daily sync flows

The team needed to create 3 data-sync flows daily (MySQL to AE, ClickHouse to AE, Databricks to AE), with each flow taking 30 minutes on average to configure manually. With the AE DataOps Skill, the system first discovered spaces with +list_spaces, then created DEV flows, added SQL task nodes, and configured dependencies and CRON (0 0 2 * * ?), and preview-published to PROD. The 3 flows went from 90 minutes of manual configuration to 15 minutes of batch command-line execution, and the CRON format error rate dropped from 40% to zero.

FAQ

What is the difference between DEV and PROD?

DEV is the development environment, responsible for creating/modifying tasks and configuring schedules; PROD is the production environment, responsible for formal execution and ops troubleshooting. All creation and modification happen in DEV and are published to PROD via preview.

Why is the CRON format 6 fields?

AE DataOps uses a 6-field format of second minute hour day month weekday, one more "second" field than the standard 5. A common template like `0 0 2 * * ?` means 2 AM daily.

What is the difference from ae-analysis?

ae-analysis handles analysis-side operations (reports/dashboards/ad-hoc queries), while ae-dataops handles data-development-side operations (flow orchestration/SQL development/data sync). The former queries data, the latter builds it.