02-Consume¶
¶
Subscribe database basic
1. Create database with tables
2. Create topics from database
3. Test database-level subscription
4. Verify all tables data consumption
5. Clean up environment
¶
Subscribe database0
1. Create database with multiple tables
2. Create database-level topics
3. Test consumption with replica settings
4. Verify data consistency across replicas
5. Clean up environment
¶
Subscribe database1
1. Create database with normal and super tables
2. Test database subscription with mixed table types
3. Verify consumption of all table types
4. Test data integrity
5. Clean up environment
¶
Subscribe database2
1. Create database with vgroups configuration
2. Test consumption across multiple vgroups
3. Verify data distribution and consumption
4. Test concurrent consumption
5. Clean up environment
¶
Subscribe database3
1. Create database with complex schema
2. Test subscription with schema changes
3. Verify consumption after DDL operations
4. Test data consistency
5. Clean up environment
¶
Subscribe database4
1. Create database with specific configurations
2. Test advanced consumption scenarios
3. Verify performance and reliability
4. Test error handling
5. Clean up environment
¶
Subscribe stable basic
1. Create database and stable
2. Create topics from stable
3. Test basic consumption functionality
4. Verify consumed data correctness
5. Clean up environment
¶
Subscribe stable0
1. Create database and stable
2. Create multiple topics from stable
3. Insert consume info to consume processor
4. Start consume processor
5. Check consume result
6. Clean up environment
¶
Subscribe stable1
1. Create database and stable
2. Create topics from stable with manual commit
3. Test auto commit functionality
4. Insert consume info to consume processor
5. Start consume processor
6. Check consume result
7. Clean up environment
¶
Subscribe stable2
1. Create database and stable
2. Create topics with manual commit disabled
3. Test multiple consumers with different commit modes
4. Insert data during consumption
5. Verify data consistency across consumers
6. Clean up environment
¶
Subscribe stable3
1. Create database and stable
2. Test multiple complex consumption scenarios
3. Test concurrent consumption with multiple threads
4. Verify offset management and commit behavior
5. Insert data during multiple consumer operations
6. Check data consistency and completeness
7. Clean up environment
¶
Subscribe stable4
1. Create database and stable
2. Test complex consumption scenarios
3. Verify data consumption with filters
4. Test multiple topic subscriptions
5. Clean up environment
¶
Advanced: alter schema
1. Create topic and subscribe
2. Start consumption
3. Execute ALTER TABLE ADD COLUMN
4. Execute ALTER TABLE DROP COLUMN
5. Execute ALTER TABLE MODIFY COLUMN
6. Verify consumption continues
7. Check schema compatibility
¶
Advanced: auto create table
1. Configure auto-create-table parameter
2. Subscribe with auto-create enabled
3. Consume data with new table definitions
4. Verify tables auto-created
5. Check table schema correct
6. Validate data inserted
¶
Consumer basic
1. Create stable and child tables, insert data
2. Create topic from stable and child table
3. Create consume info table and consume result table
4. Start consume processor
5. Check consume result
6. Negative test: create topic with wrong sql
7. Clean up environment
¶
Basic: error handling
1. Test Alter table tag in tmq batch mode, verify that tag alterations are correctly captured and reflected in the TMQ topic.
2. Test Alter table tag with filter in tmq batch mode, verify that only the tables matching the filter criteria are altered and reflected in the TMQ topic.
3. Test Alter table tag in tmq batch mode with database-level topic, verify that alterations to any table in the database are captured and reflected in the TMQ topic.
¶
Consumer bugs
1. Jira TD-31283:
- Tmq consumer fails to consume data from topic created on stable with tags after altering stable to add columns.
2. Jira TD-30270:
- Test tmq consumer subscribe/unsubscribe operations can be called multiple times.
3. Jira TD-32187:
- Test tmq consumption with meta topic on database containing stable with special tag names.
4. Jira TD-33225:
- Test tmq consumption after altering stable column compression and creating index.
5. Jira TD-32471:
- Test tmq consumption after altering stable to add new columns.
6. Jira TD-32526:
- Test tmq consumption with native C client after altering stable to add columns.
7. Jira TD-33504:
- Test tmq consumer can switch topics after unsubscribe.
8. Jira TD-35698:
- Test tmq consumption with meta topic containing decimal columns.
9. Issue 6995592605:
- Test tmq JSON meta tagName/value pairing when INSERT tag
binding order differs from the STABLE TAGS schema order.
10. Jira TD-37436:
- Test tmq consumption with meta topic on database containing stream.
11. Jira TD-38404:
- Tmq_get_json_meta behaves unexpectedly when the tags of subscribed meta messages contain empty strings.
12. Jira TS-4563:
- Test tmq consumption of unordered data inserted by stmt.
13. Jira TS-5466:
- Test tmq consumption with meta topic after altering stable to add many columns.
14. Jira TS-5906:
- Test tmq consumption after altering child table tags and inserting new data.
15. Jira TS-6115:
- Test tmq consumption with large amount of data inserted by taosBenchmark.
16. Jira TS-6392:
- Test tmq consumer group rebalance and recovery after dnode restart with WAL retention.
17. Jira TS-7402:
- Test tmq consumption with meta topic created on stable with filter conditions.
18. Jira TS-7662:
- Test tmq consumption with meta topic created on stable with where clause.
19. Jira TS-4674:
- Test tmq consumption behavior during vgroup leader rebalance in multi-replica environment.
¶
Consume: data verification
1. Insert known test data patterns
2. Create TMQ subscription
3. Consume and record all data
4. Compare consumed data with source
5. Verify data integrity (no loss, no duplicate)
¶
Consume: database verification
1. Create database with multiple tables
2. Insert test data to all tables
3. Subscribe database topic
4. Consume all data
5. Verify each table's data integrity
¶
Consumer one: from snapshot
test scenario, please refer to https://jira.taosdata.com:18090/pages/viewpage.action?pageId=135120406, firstly insert data, then start consume
1. basic1.sim: vgroups=1, one topic for one consumer
2. basic2.sim: vgroups=1, multi topics for one consumer
3. basic3.sim: vgroups=4, one topic for one consumer
4. basic4.sim: vgroups=4, multi topics for one consumer
5. snapshot1.sim: vgroups=1, multi topics for one consumer, consume from snapshot
¶
Consumer one: vgroups=1 topics=1
test scenario, please refer to https://jira.taosdata.com:18090/pages/viewpage.action?pageId=135120406, firstly insert data, then start consume
1. basic1.sim: vgroups=1, one topic for one consumer
2. basic2.sim: vgroups=1, multi topics for one consumer
3. basic3.sim: vgroups=4, one topic for one consumer
4. basic4.sim: vgroups=4, multi topics for one consumer
5. snapshot1.sim: vgroups=1, multi topics for one consumer, consume from snapshot
notes1: Scalar function: ABS/ACOS/ASIN/ATAN/CEIL/COS/FLOOR/LOG/POW/ROUND/SIN/SQRT/TAN
The above use cases are combined with where filter conditions, such as: where ts > "2017-08-12 18:25:58.128Z" and sin(a) > 0.5;
notes2: not support aggregate functions(such as sum/count/min/max) and time-windows(interval).
¶
Consumer one: vgroups=1 topics=n
test scenario, please refer to https://jira.taosdata.com:18090/pages/viewpage.action?pageId=135120406, firstly insert data, then start consume
1. basic1.sim: vgroups=1, one topic for one consumer
2. basic2.sim: vgroups=1, multi topics for one consumer
3. basic3.sim: vgroups=4, one topic for one consumer
4. basic4.sim: vgroups=4, multi topics for one consumer
5. snapshot1.sim: vgroups=1, multi topics for one consumer, consume from snapshot
notes1: Scalar function: ABS/ACOS/ASIN/ATAN/CEIL/COS/FLOOR/LOG/POW/ROUND/SIN/SQRT/TAN
The above use cases are combined with where filter conditions, such as: where ts > "2017-08-12 18:25:58.128Z" and sin(a) > 0.5;
notes2: not support aggregate functions(such as sum/count/min/max) and time-windows(interval).
¶
Consumer one: vgroups=4 topics=1
test scenario, please refer to https://jira.taosdata.com:18090/pages/viewpage.action?pageId=135120406, firstly insert data, then start consume
1. basic1.sim: vgroups=1, one topic for one consumer
2. basic2.sim: vgroups=1, multi topics for one consumer
3. basic3.sim: vgroups=4, one topic for one consumer
4. basic4.sim: vgroups=4, multi topics for one consumer
5. snapshot1.sim: vgroups=1, multi topics for one consumer, consume from snapshot
notes1: Scalar function: ABS/ACOS/ASIN/ATAN/CEIL/COS/FLOOR/LOG/POW/ROUND/SIN/SQRT/TAN
The above use cases are combined with where filter conditions, such as: where ts > "2017-08-12 18:25:58.128Z" and sin(a) > 0.5;
notes2: not support aggregate functions(such as sum/count/min/max) and time-windows(interval).
¶
Consumer one: vgroups=4 topics=n
test scenario, please refer to https://jira.taosdata.com:18090/pages/viewpage.action?pageId=135120406, firstly insert data, then start consume
1. basic1.sim: vgroups=1, one topic for one consumer
2. basic2.sim: vgroups=1, multi topics for one consumer
3. basic3.sim: vgroups=4, one topic for one consumer
4. basic4.sim: vgroups=4, multi topics for one consumer
5. snapshot1.sim: vgroups=1, multi topics for one consumer, consume from snapshot
notes1: Scalar function: ABS/ACOS/ASIN/ATAN/CEIL/COS/FLOOR/LOG/POW/ROUND/SIN/SQRT/TAN
The above use cases are combined with where filter conditions, such as: where ts > "2017-08-12 18:25:58.128Z" and sin(a) > 0.5;
notes2: not support aggregate functions(such as sum/count/min/max) and time-windows(interval).
¶
Consumer two: from snapshot
test scenario, please refer to https://jira.taosdata.com:18090/pages/viewpage.action?pageId=135120406, firstly insert data, then start consume
1. basic1Of2Cons.sim: vgroups=1, one topic for 2 consumers
2. basic2Of2Cons.sim: vgroups=1, multi topics for 2 consumers
3. basic3Of2Cons.sim: vgroups=4, one topic for 2 consumers
4. basic4Of2Cons.sim: vgroups=4, multi topics for 2 consumers
5. basic2Of2ConsOverlap.sim: vgroups=1, multi topics for 2 consumers, the topics consumed by consumers overlap with each other
6. snapshot.sim: vgroups=1, multi topics for 2 consumers, consume from snapshot
notes1: Scalar function: ABS/ACOS/ASIN/ATAN/CEIL/COS/FLOOR/LOG/POW/ROUND/SIN/SQRT/TAN
The above use cases are combined with where filter conditions, such as: where ts > "2017-08-12 18:25:58.128Z" and sin(a) > 0.5;
notes2: not support aggregate functions(such as sum/count/min/max) and time-windows(interval).
¶
Consumer two: from snapshot
test scenario, please refer to https://jira.taosdata.com:18090/pages/viewpage.action?pageId=135120406, firstly insert data, then start consume
1. basic1Of2Cons.sim: vgroups=1, one topic for 2 consumers
2. basic2Of2Cons.sim: vgroups=1, multi topics for 2 consumers
3. basic3Of2Cons.sim: vgroups=4, one topic for 2 consumers
4. basic4Of2Cons.sim: vgroups=4, multi topics for 2 consumers
5. basic2Of2ConsOverlap.sim: vgroups=1, multi topics for 2 consumers, the topics consumed by consumers overlap with each other
6. snapshot.sim: vgroups=1, multi topics for 2 consumers, consume from snapshot
¶
Consumer two: vgroups=1 topics=1
test scenario, please refer to https://jira.taosdata.com:18090/pages/viewpage.action?pageId=135120406, firstly insert data, then start consume
1. basic1Of2Cons.sim: vgroups=1, one topic for 2 consumers
2. basic2Of2Cons.sim: vgroups=1, multi topics for 2 consumers
3. basic3Of2Cons.sim: vgroups=4, one topic for 2 consumers
4. basic4Of2Cons.sim: vgroups=4, multi topics for 2 consumers
5. basic2Of2ConsOverlap.sim: vgroups=1, multi topics for 2 consumers, the topics consumed by consumers overlap with each other
6. snapshot.sim: vgroups=1, multi topics for 2 consumers, consume from snapshot
notes1: Scalar function: ABS/ACOS/ASIN/ATAN/CEIL/COS/FLOOR/LOG/POW/ROUND/SIN/SQRT/TAN
The above use cases are combined with where filter conditions, such as: where ts > "2017-08-12 18:25:58.128Z" and sin(a) > 0.5;
notes2: not support aggregate functions(such as sum/count/min/max) and time-windows(interval).
¶
Consumer two: vgroups=1 topics=n
test scenario, please refer to https://jira.taosdata.com:18090/pages/viewpage.action?pageId=135120406, firstly insert data, then start consume
1. basic1Of2Cons.sim: vgroups=1, one topic for 2 consumers
2. basic2Of2Cons.sim: vgroups=1, multi topics for 2 consumers
3. basic3Of2Cons.sim: vgroups=4, one topic for 2 consumers
4. basic4Of2Cons.sim: vgroups=4, multi topics for 2 consumers
5. basic2Of2ConsOverlap.sim: vgroups=1, multi topics for 2 consumers, the topics consumed by consumers overlap with each other
6. snapshot.sim: vgroups=1, multi topics for 2 consumers, consume from snapshot
notes1: Scalar function: ABS/ACOS/ASIN/ATAN/CEIL/COS/FLOOR/LOG/POW/ROUND/SIN/SQRT/TAN
The above use cases are combined with where filter conditions, such as: where ts > "2017-08-12 18:25:58.128Z" and sin(a) > 0.5;
notes2: not support aggregate functions(such as sum/count/min/max) and time-windows(interval).
¶
Consumer two: vgroups=4 topics=1
test scenario, please refer to https://jira.taosdata.com:18090/pages/viewpage.action?pageId=135120406, firstly insert data, then start consume
1. basic1Of2Cons.sim: vgroups=1, one topic for 2 consumers
2. basic2Of2Cons.sim: vgroups=1, multi topics for 2 consumers
3. basic3Of2Cons.sim: vgroups=4, one topic for 2 consumers
4. basic4Of2Cons.sim: vgroups=4, multi topics for 2 consumers
5. basic2Of2ConsOverlap.sim: vgroups=1, multi topics for 2 consumers, the topics consumed by consumers overlap with each other
6. snapshot.sim: vgroups=1, multi topics for 2 consumers, consume from snapshot
notes1: Scalar function: ABS/ACOS/ASIN/ATAN/CEIL/COS/FLOOR/LOG/POW/ROUND/SIN/SQRT/TAN
The above use cases are combined with where filter conditions, such as: where ts > "2017-08-12 18:25:58.128Z" and sin(a) > 0.5;
notes2: not support aggregate functions(such as sum/count/min/max) and time-windows(interval).
¶
Consumer two: vgroups=4 topics=n
test scenario, please refer to https://jira.taosdata.com:18090/pages/viewpage.action?pageId=135120406, firstly insert data, then start consume
1. basic1Of2Cons.sim: vgroups=1, one topic for 2 consumers
2. basic2Of2Cons.sim: vgroups=1, multi topics for 2 consumers
3. basic3Of2Cons.sim: vgroups=4, one topic for 2 consumers
4. basic4Of2Cons.sim: vgroups=4, multi topics for 2 consumers
5. basic2Of2ConsOverlap.sim: vgroups=1, multi topics for 2 consumers, the topics consumed by consumers overlap with each other
6. snapshot.sim: vgroups=1, multi topics for 2 consumers, consume from snapshot
notes1: Scalar function: ABS/ACOS/ASIN/ATAN/CEIL/COS/FLOOR/LOG/POW/ROUND/SIN/SQRT/TAN
The above use cases are combined with where filter conditions, such as: where ts > "2017-08-12 18:25:58.128Z" and sin(a) > 0.5;
notes2: not support aggregate functions(such as sum/count/min/max) and time-windows(interval).
¶
Consume: group
1. Create consumer group
2. Add multiple consumers to group
3. Start consumption with load balancing
4. Verify partition assignment
5. Test consumer rebalance on add/remove
¶
Advanced: data precision
1. Test millisecond precision
2. Test microsecond precision
3. Test nanosecond precision
4. Verify timestamp conversion
5. Check precision preservation
¶
Basic: database topic
1. Create database with multiple tables
2. Create database-level topic
3. Subscribe entire database
4. Insert data to various tables
5. Verify all tables data consumed
¶
Operation: delete multi ctb
1. Create multiple child tables
2. Insert data to all tables
3. Subscribe super table topic
4. DELETE rows from multiple tables
5. Consume all delete messages
6. Verify deletions from all tables
¶
Operation: delete single ctb
1. Create child table with data
2. Create topic subscription
3. Execute DELETE operations
4. Consume delete messages
5. Verify deletion propagated correctly
¶
Consume: discontinuous data
1. Insert data with discontinuous timestamps
2. Create topic subscription
3. Consume data in order
4. Verify all records received with correct timestamps
¶
Cluster: dnode restart
1. Start consumption process
2. Restart dnode during consumption
3. Wait for dnode recovery
4. Resume consumption automatically
5. Verify no data loss
6. Check offset preserved
¶
Cluster: dnode restart variant
1. Setup multi-dnode cluster
2. Start TMQ consumption
3. Restart multiple dnodes sequentially
4. Verify automatic failover
5. Check data continuity
6. Validate recovery behavior
¶
Operation: drop consumer
1. Create consumer and subscribe
2. Start consumption
3. Drop consumer connection
4. Verify consumer removed from group
5. Check offset committed
6. Verify cleanup
¶
Cluster: drop db replica3
1. Setup 3-replica cluster
2. Create database with replication
3. Start TMQ consumption
4. Drop database during consumption
5. Verify transaction conflict handling
6. Check graceful error reporting
¶
Operation: drop table snapshot
1. Create normal table with data
2. Start snapshot mode consumption
3. Drop table after some consumption
4. Verify snapshot behavior with dropped table
5. Check graceful handling
¶
Operation: drop table wal
1. Create normal table with data
2. Start WAL mode consumption
3. Drop normal table during consumption
4. Verify drop event handling
5. Check consumer handles table removal
¶
Operation: drop super table
1. Create super table with child tables
2. Start consumption
3. Drop child tables one by one
4. Drop super table
5. Verify all drop events handled
6. Check subscription cleanup
¶
Basic: error handling
1. Test invalid topic name
2. Test invalid consumer group.id
3. Test subscribe non-existent topic
4. Test invalid consumer parameters
5. Verify appropriate error messages
¶
Basic: max groups limit
1. Create database and topic
2. Create 100 consumer groups (maximum allowed)
3. Verify all groups work correctly
4. Try creating 101st group and verify limit
¶
Basic: max topics limit
1. Create database with super table
2. Create 20 topics (maximum allowed)
3. Verify topic creation success
4. Try creating 21st topic and verify error
5. Drop all topics and cleanup
¶
Advanced: metadata json
1. Create topics with metadata
2. Query metadata via JSON interface
3. Verify JSON format correctness
4. Check all metadata fields present
5. Validate JSON schema
¶
Cluster: mnode switch
1. Setup 3-mnode cluster
2. Start consumption
3. Trigger mnode leader switch
4. Continue consumption
5. Verify seamless transition
6. Check no interruption
¶
Basic: offset operations
1. Create topic and start consumption
2. Test seek operation to specific offset
3. Query current position
4. Commit offset manually
5. Restart consumer and verify offset restored
6. Call tmq_offset_test to test
¶
Basic: consumer parameters
1. Test auto.commit enabled/disabled
2. Test auto.offset.reset earliest/latest
3. Test group.id configuration
4. Verify consumer behavior with different parameters
¶
Basic: primary key support
1. Create tables with primary key
2. Insert duplicate key data
3. Create TMQ subscription on primary key table
4. Consume and verify primary key constraints
5. Test UPDATE on primary key records
¶
Advanced: raw block interface
1. Use raw block API for consumption
2. Fetch data in block format
3. Parse block structure
4. Verify block metadata
5. Check performance benefit
6. Validate block data
¶
Advanced: replay operations
1. Run replay_test executable to test TMQ replay functionality
2. Verify no errors occur during execution
3. Confirm successful completion of the test
¶
Tmq consume restart
1. create stable and topic
2. insert data
3. consume part of data and commit
4. restart taosd
5. continue consume data and check
6. kill taosd process
7. restart taosd
8. continue consume data and check
9. clean up environment
¶
Advanced: schema changes
1. Create tables and start consumption
2. Add column during consumption
3. Drop column during consumption
4. Modify column type
5. Verify schema evolution handled
6. Check backward compatibility
¶
Advanced: seek commit
1. Start consumption
2. Execute seek to offset
3. Consume from new position
4. Perform manual commit
5. Verify offset updated
6. Test commit persistence
¶
Consume: snapshot basic
1. Insert historical data to TSDB
2. Flush data to disk
3. Create topic with snapshot mode
4. Consume from TSDB snapshot
5. Verify complete historical data retrieved
¶
Consume: snapshot filter
1. Insert data to TSDB
2. Create filtered topic (SELECT with WHERE)
3. Enable snapshot consumption
4. Consume filtered historical data
5. Verify only matching records retrieved
¶
Consume: function filter
1. Insert data to child table
2. Create topic with UDF function
3. Add WHERE filter condition
4. Consume from snapshot with computation
5. Verify UDF results and filtering
¶
Consume: function filter variant
1. Setup child table with data
2. Apply complex UDF and filters
3. Enable snapshot mode
4. Consume computed results
5. Validate function output
¶
Consume: complex function filter
1. Setup multi-vgroup multi-table structure
2. Apply UDF with complex filters
3. Create snapshot topic
4. Consume computed data from all vgroups
5. Verify UDF and filter correctness
¶
Consume: complex filter variant
1. Create complex table structure
2. Apply multiple UDFs
3. Add WHERE conditions
4. Consume snapshot with computations
5. Validate complex query results
¶
Consume: multi child tables
1. Create super table with multiple child tables
2. Insert data to all child tables
3. Flush to TSDB
4. Create snapshot topic on super table
5. Consume all child tables data
6. Verify complete consumption
¶
Consume: multi ctb filter
1. Create multiple child tables
2. Insert diverse data
3. Apply filters to topic
4. Consume snapshot with filtering
5. Verify filtered results from all tables
¶
Consume: multi vgroups
1. Create database with 4+ vgroups
2. Insert data distributed across vgroups
3. Create snapshot topic
4. Consume from all vgroups simultaneously
5. Verify complete data from all vgroups
¶
Consume: multi vg filter
1. Setup multi-vgroup database
2. Distribute data across vgroups
3. Create filtered snapshot topic
4. Consume from all vgroups with filter
5. Verify filter consistency across vgroups
¶
Consume: single child table
1. Create single child table
2. Insert data and flush to TSDB
3. Create snapshot topic
4. Consume single table from snapshot
5. Verify data completeness
¶
Consume: single ctb filter
1. Create child table with data
2. Apply column and WHERE filters
3. Create snapshot topic with filter
4. Consume filtered snapshot
5. Verify filter accuracy
¶
Filter: column filter
1. Create super table with multiple columns
2. Create topic selecting specific columns
3. Insert full row data
4. Consume with column filter
5. Verify only selected columns received
¶
Filter: tag filter
1. Create super table with tag columns
2. Create child tables with different tags
3. Create topic with tag filter
4. Insert data to all child tables
5. Consume filtered by tag
6. Verify only matching tags received
¶
Filter: tag multi filter
1. Create super table with tags
2. Create multiple child tables
3. Apply complex tag filter (AND/OR)
4. Insert data to all tables
5. Consume with tag filtering
6. Verify multi-table tag filter accuracy
¶
Filter: where filter
1. Create super table with data
2. Create topic with WHERE condition
3. Insert matching and non-matching rows
4. Consume filtered data
5. Verify only matching rows received
¶
Cluster: subscribe replica3
1. Create 3-replica super table
2. Insert data to all replicas
3. Subscribe with replica awareness
4. Consume from replica set
5. Verify replica consistency
6. Check failover behavior
¶
Advanced: taosx sync
1. Configure taosx synchronization
2. Setup source and target clusters
3. Start taosx sync via TMQ
4. Insert data to source
5. Verify data synced to target
6. Check sync accuracy
¶
Basic: topics info query
1. Create database and topic
2. Query information_schema.ins_topics
3. Verify topic metadata (name, db, create_time, sql)
4. Drop topic and verify cleanup
¶
Consume: tsdb and wal
1. Insert historical data (flush to TSDB)
2. Insert new data (remain in WAL)
3. Create topic consuming both sources
4. Consume merged data stream
5. Verify correct merge and ordering
¶
Consume: tsdb wal multi
1. Create multiple child tables
2. Write historical data to TSDB
3. Write recent data to WAL
4. Subscribe all tables
5. Consume from TSDB and WAL
6. Verify complete data from both sources
¶
Advanced: udf functions
1. Create UDF functions
2. Create topic with UDF in SELECT
3. Insert source data
4. Consume computed results
5. Verify UDF executed correctly
6. Check function output
¶
Advanced: udf snapshot mode
1. Configure snapshot with UDF
2. Flush data to TSDB
3. Apply UDF to snapshot topic
4. Consume computed snapshot
5. Verify UDF on historical data
¶
Advanced: udf wal mode
1. Setup UDF in topic
2. Configure WAL mode
3. Create topic with UDF
4. Consume from WAL with computation
5. Verify UDF results from WAL
¶
Operation: update single ctb
1. Create child table with initial data
2. Create topic subscription
3. Execute UPDATE on existing rows
4. Consume update messages
5. Verify updated values received
¶
Operation: update snapshot mode
1. Configure snapshot mode (snapshot=1)
2. Flush data to TSDB
3. Execute UPDATE on historical data
4. Consume from snapshot + WAL
5. Verify snapshot mode update behavior
¶
Consume: update tablelist for tmq
1. Insert data to TSDB
2. Create filtered topic (SELECT with WHERE)
3. Enable snapshot consumption
4. Consume filtered historical data
5. Verify only matching records retrieved
¶
Operation: update wal mode
1. Configure WAL mode (snapshot=0)
2. Create subscription
3. Execute UPDATE operations
4. Consume from WAL only
5. Verify WAL-based update handling
¶
Operation: update while consume
1. Start consumption process
2. Concurrently execute UPDATE operations
3. Consume updates in real-time
4. Verify no data loss during concurrent updates
¶
Cluster: vnode replicate
1. Setup replica configuration
2. Create replicated vgroups
3. Start consumption
4. Verify replica synchronization
5. Test consumption from different replicas
6. Check consistency
¶
Cluster: vnode split column
1. Create column-based subscription
2. Start consumption
3. Trigger vnode split operation
4. Continue consumption after split
5. Verify no data loss during split
6. Check vnode rebalancing
¶
Cluster: vnode split db
1. Create database subscription
2. Insert data distributed across vnodes
3. Trigger vnode split
4. Continue database-level consumption
5. Verify all vnode data consumed
6. Check integrity after split
¶
Cluster: vnode split db-nowal
1. Setup database topic
2. Configure WAL retention policy
3. Execute vnode split
4. Verify WAL behavior
5. Continue consumption
6. Check data completeness
¶
Cluster: vnode split dup-nowal
1. Setup duplicate data scenario
2. Configure WAL retention
3. Split vnodes
4. Verify duplicate handling with WAL
5. Check data accuracy
¶
Cluster: vnode split duplicate
1. Insert potentially duplicate data
2. Execute vnode split
3. Consume all data
4. Verify deduplication works
5. Check no duplicates after split
¶
Cluster: vnode split no-remove
1. Setup vnode split test
2. Configure no WAL removal
3. Execute vnode split
4. Verify WAL retained
5. Continue consumption
6. Check data continuity
¶
Cluster: vnode split ntb
1. Create normal table subscription
2. Insert data
3. Trigger vnode split
4. Continue consumption
5. Verify normal table unaffected
6. Check data integrity
¶
Cluster: vnode split no-wal
1. Create SELECT topic
2. Configure no WAL removal
3. Split vnodes
4. Continue query consumption
5. Verify query continuity
6. Check WAL retention effect
¶
Cluster: vnode split stb
1. Create super table subscription
2. Distribute child tables across vnodes
3. Execute vnode split
4. Resume consumption
5. Verify all child tables data
6. Check split handling
¶
Cluster: vnode split stb no-wal
1. Setup super table topic
2. Configure WAL retention
3. Split vnodes
4. Verify WAL behavior
5. Continue consumption
6. Check data completeness
¶
Cluster: vnode split select
1. Create SELECT-based topic on super table
2. Insert data with computation
3. Trigger vnode split
4. Continue consuming query results
5. Verify computed data correct after split
¶
Cluster: vnode transform db
1. Create database topic
2. Start consumption
3. Execute vnode transform (rebalance)
4. Continue consumption
5. Verify data migration complete
6. Check no data loss
¶
Cluster: vnode transform db-wal
1. Setup database topic
2. Transform vnodes with WAL removal
3. Consume during transformation
4. Verify WAL cleanup
5. Check data integrity
¶
Cluster: vnode transform stb
1. Create super table topic
2. Start consumption
3. Execute vnode transformation
4. Resume consumption
5. Verify transformation handled
6. Check data consistency
¶
Cluster: vnode transform stb-wal
1. Setup super table topic
2. Transform with WAL removal
3. Continue consumption
4. Verify WAL behavior
5. Check data completeness
¶
summary: test tmq vtable
description:
snapshot = true, stable topic / db topic, check get_raw/tmq_get_json
snapshot = false, stable topic / db topic, check get_raw/write_raw
ref is from another db table col, virtual_normal_table, virtual_child_table
ref is all, or ref is partial
create vtable, create normal v table.
add col for normal v table
alter ref for child vtable
¶
Advanced: wal removal
1. Configure WAL retention period
2. Insert data generating WAL
3. Trigger WAL cleanup
4. Verify old WAL removed
5. Check space reclaimed
6. Ensure consumption not affected